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Abstract 

With  respect  to  military  applications,  classification  systems  are  employed  to 
remotely  assess  whether  an  element  of  interest  falls  into  a  “target”  class  or  “non-target” 
class.  The  name  of  the  classes  is  arbitrary,  but  the  role  of  the  classification  system 
remains  the  same.  These  systems  also  have  uses  in  fields  as  far  ranging  as  biostatistics  to 
search  engine  keyword  analysis.  The  performance  of  the  system  is  often  summarized  as  a 
trade-off  between  the  proportions  of  elements  correctly  labeled  as  “target”  plotted  against 
the  number  of  elements  incorrectly  labeled  as  “target.”  The  first  term  is  generally 
regarded  as  the  “hit  rate”  and  the  second  term  as  the  “false  alarm  rate.”  These  are 
empirical  estimates  of  the  true  positive  and  false  positive  rates.  These  rates  are  often 
plotted  to  create  a  receiver  operating  characteristic  (ROC)  curve  that  acts  as  a  visual  tool 
to  assess  classification  system  performance.  The  performance  of  the  system(s)  can 
possibly  be  increased  if  the  information  provided  by  both  systems  can  be  fused  together 
to  create  a  new,  combined  system  using  any  number  of  techniques.  The  research 
contained  in  this  thesis  focuses  specifically  on  the  label  fusion  technique  and  the  bias  that 
can  occur  when  using  incorrect  assumptions  regarding  the  partitioning  of  the  event  set. 
This  partitioning  may  be  defined  in  terms  of  what  will  be  called  within  and  across  label 
fusion.  The  major  goals  of  this  thesis  are  the  formulaic  development  and  quantification  of 
performance  bias  between  different  types  of  across  and  within  label  fusion  and  analysis 
of  the  effects  of  individual  classification  system  performance,  correlation,  and  target 
environment  on  the  magnitude  of  bias  between  these  two  types  of  label  fusion.  Formulas 
developed  may  be  used  to  adjust  optimal  parameters  and  performance  measures  to 
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appropriately  reflect  fused  system  performance  on  various  platforms.  Thus,  this  research 
can  be  applied  in  the  future  to  address  the  inherent  bias  that  may  be  built  into  fused 
classification  systems. 


QUANTIFYING  PERFORMANCE  BIAS  IN  LABEL  FUSION 

I  Introduction 

1.1  Background 

As  technology  advances  and  access  to  inexpensive  and  efficient  computing 
resources  continues  to  rise,  there  is  reason  to  believe  that  sole  reliance  on  the  human 
element  in  any  number  of  decision-based  applications  will  be  reduced  and  there  will  be 
an  increased  use  of  automated  systems.  Such  technology  has  advantages.  For  example,  if 
a  machine  can  take  the  place  of  a  human  being  in  a  potentially  life  threatening  situation, 
why  not  rely  on  autonomous  processes?  However,  as  research  into  the  area  of  decision 
and  classification  system  theory  has  evolved,  it  has  become  clear  that  the  process  of 
human-based  decision  making  is  a  difficult  task  for  a  machine.  Most  notably,  there  are  a 
considerable  number  of  ways  to  analyze  data  and  make  a  decision. 

Classically,  the  goal  of  label  fusion  is  to  combine  the  output  from  multiple 
classification  systems  to  improve  predictive  accuracy.  This  makes  intuitive  sense  from 
the  point  of  view  that  different  classification  systems  excel  at  classifying  different  events. 
Further,  the  performance  of  different  combinations  of  classification  systems  depends  in 
large  part  on  the  fusion  rule  to  be  used.  There  are  times,  however,  that  fusing  multiple 
classification  systems  together  can  actually  decrease  performance  and  in  this  instance,  it 
may  be  best  to  not  fuse  and  instead  choose  the  optimally  performing  individual 
classification  system. 

Apart  from  fusion  rule  and  choice  of  classification  systems,  the  level  of 
classification  system  dependence  is  a  necessary  consideration.  Some  methods  make  no 
assumption  about  the  dependence  of  individual  classification  systems  while  others 
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inherently  assume  independence.  For  example,  probabilistic  neural  networks  make  no 
assumptions  regarding  the  level  of  dependence  between  individual  classification  systems 
{Leap  et  al:  2008).  Instead,  the  probabilistic  neural  network  is  tuned  to  the  targets  it  is 
designed  to  classify  by  using  a  portion  of  the  available  data  as  training  material.  The 
theory  that  this  thesis  is  based  upon,  label  fusion,  requires  that  the  individual 
classification  systems  are  independent  in  order  to  derive  the  functions  for  the  receiver 
operating  characteristic  (ROC)  curve.  The  ROC  curve  is  a  graphical  tool  for  depicting 
true  hit  rate  along  the  vertical  axis  (the  number  of  target  events  correctly  classified  as 
targets,  i.e.,  true  positive  rate)  as  compared  to  false  alarm  rate  along  the  horizontal  axis 
(the  number  of  target  events  incorrectly  classified  as  non-targets,  i.e.,  false  positive  rate). 

With  the  notions  of  fusion  and  classification  system  dependence  in  place,  the 
focus  becomes  how  to  optimize  performance  given  the  systems  and  fusion  rules 
available.  It  is  common  to  apply  a  series  of  logical  rules  and  combinations  of  logical  rules 
and  choose  the  combination  that  optimizes  performance  {W.  Khreich  et  al:  2012). 
Another  common  method  is  to  treat  the  output  of  the  individual  classification  systems  as 
input  to  a  neural  network  and  run  a  regression-based  analysis  {Leap  et  al:  2008).  Along 
with  optimization  of  performance  comes  the  consideration  of  optimization  of  available 
resources.  Fusing  classification  systems  tuned  to  different  target  types,  perhaps  from 
legacy  systems,  invokes  concerns  regarding  the  probability  with  respect  to  target 
prevalence  and  event  set  partitions.  However,  systems  combined  using  label  fusion  rules 
are  especially  prone  to  simplifying  assumptions  which  may  overlook  target  partitions. 
Mathematical  and  computational  modeling  generally  performs  well  in  this  regard  relative 
to  diagnostic  testing  or  actual  engineering  testing.  As  such,  the  fused  system  should 
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outperform  any  individual  system  and  perform  efficiently  despite  environmental 
constraints  such  as  target  prevalence  and  classification  system  correlation. 

1.2  Problem  Statement 

The  research  contained  herein  addresses  the  quantification  of  bias  between  what 
will  be  defined  as  across  and  within  label-fused  classification  systems  and  the  effects  of 
individual  classification  system  performance,  correlation,  and  target  environment  on  such 
bias.  Specifically,  fused  systems  that  do  not  account  for  differences  in  the  partitioning  of 
targets  between  fused  systems  produce  errors,  or  a  bias,  in  system  performance. 
Depending  on  how  the  specific  target  partitions  of  the  event  set  are  defined  for  the 
classification  systems  to  be  fused  gives  rise  to  the  notion  of  within  versus  across  label 
fusion.  It  is  in  mistaking  between  what  can  be  labeled  as  within  and  across  fusion  that  the 
bias  in  system  performance  occurs.  Thus,  this  research  involves  the  derivation  of 
formulas  that  allow  for  quick  and  easily  implementable  bias  correction  algorithms. 
Building  on  the  work  of  (Schubert:  2005),  (Leap  et  al:  2008),  (Schubert  et  al:  2005), 
(Oxley,  Bauer:  2003),  who  have  previously  investigated  and  derived  functions  for  the 
ROC  curves  of  label-fused  systems,  the  formulas  for  bias  will  be  derived  using  these 
formulas.  The  algorithm  developed  here  will  create  a  graphical  and  analytical  tool  for 
measuring  bias  in  the  form  of  a  bias  curve.  Using  the  work  of  (Schubert:  2005), 
correlation  between  classification  systems  will  be  examined.  Individual  system 
performance  and  structure  of  the  target  environment  is  also  considered. 

1.3  Research  Methodology 

Prior  to  developing  a  functional  form  for  the  bias,  some  rudimentary  classification 
system  theory  and  label  fusion  theory  must  be  derived  along  with  the  functions  for  ROC 
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curves  under  both  Boolean  AND/OR  rules  and  within  and  across  label  fusion.  The  theory 
of  correlation  under  these  Boolean  rules  will  be  discussed  briefly.  With  the  ROC  curves, 
the  bias  can  be  defined  as  the  difference  in  true  positive  rate  for  two  ROC  curves  under 
different  label  fusion  rules.  These  formulas  have  been  derived  under  the  assumption  of 
conditional  independence  with  respect  to  the  non-target  partition  of  the  event  set.  This 
will  have  important  implications  when  it  comes  to  assessing  the  correlation  between 
different  label-fused  classification  systems.  A  specific  bias  function  is  derived  for 
comparing  each  type  of  across  versus  within  label-fused  system  under  a  given  Boolean 
rule.  Using  simulations,  the  performance  of  classification  systems  in  different  target 
environments  under  different  assumptions  of  correlation  and  the  bias  between  different 
combined  label-fused  systems  are  compared. 

1.4  Assumptions 

The  main  assumptions  made  in  this  document  have  been  largely  inherited  from 
previous  research.  First,  it  is  assumed  that  there  is  a  two-class  label  set.  From  the  point  of 
view  of  military  application,  the  two-class  label  set  may  generally  be  composed  of  the 
target  class  and  non-target  class,  though  it  need  not  be.  Consider  a  simple  example 
involving  a  two-class  label  set.  Suppose  a  classification  system  is  built  to  classify  a 
specific  type  of  radar  signature  for  a  type  of  enemy  aircraft.  This  system  could  employ  a 
vector  of  probability  values  that  favors  (that  is,  employs  a  higher  weighting  of 
probability)  the  classification  of  high  velocity  fighter  jets.  Perhaps  other,  less  highly 
weighted  parameters  are  concerned  with  classifying  support  aircraft.  Now  suppose  that  an 
array  of  these  systems  is  put  out  into  the  field  and  a  high  velocity  aircraft  is  correctly 
classified  as  a  target.  After  a  series  of  aircraft  have  been  classified  by  the  systems,  the 
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performance  of  the  array  can  be  depicted  graphically  using  the  ROC  curve  where  true 
positive  or  hit  rate  is  plotted  along  the  vertical  axis  and  the  false  positive  or  false  alarm 
rate  is  plotted  along  the  horizontal  axis. 

Notice  that  the  classification  systems  themselves  did  not  identify  the  actual 
aircraft,  but  rather  used  classification  systems  to  classify  the  aircraft  into  a  specific  class, 
e.g.,  a  target.  This  is  a  subtlety  that  is  inherent  with  classification  system  fusion:  the 
classification  systems  cannot  identify  the  elements  of  interest,  only  assign  a  class  label 
determined  by  pre-defined  thresholds.  This  loss  of  information  is  one  disadvantage 
inherent  to  label  based  fusion. 

The  second  assumption  inherited  from  prior  research  is  that  the  classification 
systems  are  combined  using  label  fusion.  In  label  fusion,  the  fusion  of  classification 
systems  occurs  after  both  systems  have  produced  their  own  label  sets.  These  label  sets  are 
then  combined  using  some  combination  of  logical  Boolean  rules  or  other  rules  to  produce 
a  combined  label  set.  There  are  other  methods  of  fusion.  Another  type  of  fusion  is  feature 
fusion,  where  a  decision  is  made  with  respect  to  analysis  of  target  features  (i.e.  Does  the 
vehicle  have  wheels  or  tracks?)  and  a  decision  is  made  using  data  from  both  systems. 
With  respect  to  Boolean  rules,  only  the  logical  AND/OR  rules  will  be  investigated  in  this 
thesis.  The  derivation  of  ROC  curves  and  therefore  the  derivation  of  bias  formulas  are 
done  with  respect  to  label-fusion  under  these  two  rules.  This  does  limit  the  concept  of 
bias  currently  to  single  applications  of  Boolean  rules,  but  considering  they  are  easy  to 
implement  and  used  heavily  in  research,  this  limitation  is  minimal. 

A  third  assumption  is  the  notion  of  conditional  independence  of  classification  systems 
with  respect  to  the  non-target  partition  of  the  event  set.  ROC  curve  formulas  for  the  fused 
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classification  system  had  been  developed  previously  in  a  manner  such  that  the  formula 
could  be  written  in  terms  of  the  ROC  curves  of  the  individual  classification  systems.  In 
this  manner,  the  performance  of  the  combined  system  could  be  determined  from  the 
performances  of  the  individual  systems.  Hence,  the  need  for  further  testing  is  reduced  and 
the  use  of  previous  testing  results,  as  from  legacy  data  sets,  is  leveraged. 

1.5  Research  Implications 

The  results  from  this  research  may  be  used  to  determine  the  difference  in  classification 
system  performance  between  competing  label-based  fusion  techniques  in  application. 
Most  importantly,  it  may  draw  attention  to  the  importance  of  choosing  the  correct  type  of 
label  fusion  rule  when  the  partitioning  of  events  is  finer  than  either  target  type  or  non¬ 
target  type.  This  has  not  been  scrutinized  closely  by  many  active  researchers.  Bias 
formulas  may  be  used  to  adjust  previously  fused  classification  systems  to  a  corrected 
performance  by  reweighting  target  prevalence,  adjusting  target  environment,  and 
adjusting  the  level  of  correlation  between  individual  classification  systems.  This  is 
accomplished  through  a  simple  and  cost  effective  way  of  simulating  the  performance  of 
classification  systems  a  computer  program.  The  groundwork  may  also  be  expanded  upon 
by  other  researchers  to  develop  techniques  for  comparing  the  performance  of  systems  that 
are  not  based  on  Boolean  rules  as  well  as  relaxing  the  notion  of  conditional  independence 
of  classification  systems  with  respect  to  the  non-target  partition. 

1.6  Outline  of  Sections 

In  section  II,  a  brief  literature  review  outlines  the  research  conducted  into  the  area 
of  label  based  fusion  up  to  the  current  point  in  time.  In  section  III,  the  underlying  theory 
of  classification  system  fusion  and  across  and  within  label  based  fusion  is  derived.  In 
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section  IV,  a  brief  outline  of  the  computer  simulation  will  be  given.  Finally,  in  section  V 
the  formulas  for  bias  between  competing  within  and  across  label-fused  systems  is  derived 
and  the  simulation  data  is  analyzed. 
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II  Literature  Review 


As  budgets  tighten  and  access  to  relatively  inexpensive  and  efficient  computing 
resources  continues  to  rise,  researchers  are  continuously  searching  for  ways  to  improve 
the  performance  of  autonomous  classification  systems.  Generally  the  classification 
system  being  utilized  is  composed  of  an  ensemble  of  classification  systems  that  is  then 
fused  together  using  some  sort  of  decision  rule.  As  outlined  in  the  introduction  of 
( Liggins  et  al:  1997),  “fusion  is  necessary  to  integrate  the  data  from  different  sensors  and 
extract  the  relevant  information  on  the  targets.”  Traditionally,  fusion  occurred  in  a 
centralized  architecture.  That  is,  data  from  various  sources  were  sent  to  a  single  location 
where  the  data  was  then  fused  in  some  way.  Eventually  this  architecture  evolved  into  a 
hierarchy  of  classification  systems  where  low  level  systems  process  data  and  then  send 
this  information  to  more  specialized  classification  systems  to  improve  accuracy  of 
decision.  The  means  and  manner  in  which  information  is  being  fused  is  rapidly  changing 
to  include  a  multitude  of  application  specific  methods. 

There  are  many  areas  of  active  research  where  the  fusion  of  information  takes  a 
lead  role.  For  example,  one  of  the  more  active  areas  of  research  involves  the  testing  of 
hypotheses,  and  more  specifically,  either  the  identification  or  classification  of  an  element 
of  interest.  In  classification,  information  is  fused  to  put  an  element  of  interest  into  one  of 
a  series  of  classes  where  common  features  are  shared.  Human  face  recognition  and 
gender  recognition  are  two  areas  where  automated  machine  classification  is  of  current 
interest  (7.  Pang  et  al:  2012).  Identification  is  a  more  specific  form  of  classification 
where  the  element  can  be  physically  labeled  with  regard  to  its  true  identity.  Another 
example  of  active  research  with  regard  to  information  fusion  is  automatic  target 
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recognition  (ATR).  In  ATR  research,  the  goal  is  to  develop  methods  which  permit  the 
tracking  and  classification  of  objects  through  time  and  space.  This  has  clear  military  and 
civilian  applications,  but  is  not  the  primary  subject  of  this  thesis.  Further  information  on 
the  subject  can  be  gleaned  from  any  number  of  sources  such  as  ( Pilcher  and  Khotanzad: 
2008),  (Gross:  1999),  and  (Padgett  and  Woodward:  1997). 

2. 1  Fusion  Methods 

There  are  a  number  of  ways  to  fuse  classification  systems.  Among  the  more 
prevalent  methods  in  modem  research  are  the  use  of  neural  networks,  Boolean  rules  or 
voting  rules,  and  statistical  methods. 

The  fusion  of  classification  systems  or  classification  systems  tends  to  occur  most 
commonly  on  either  the  features  of  the  data  or  on  the  labels  produced  by  the  system.  In 
feature  fusion,  classification  systems  make  decisions  by  analyzing  the  distinguishing 
attributes  of  those  elements  in  question.  In  label  fusion,  the  fusing  of  information  occurs 
after  the  individual  classification  systems  have  already  given  a  class  label  to  the  element 
in  question.  These  two  fusion  approaches  are  arguably  the  most  common  in  literature. 

The  use  of  neural  networks  is  ubiquitous  in  literature  (Sinha,  Gupta,  Rao:  2001), 
(Sani  et  al:  2009),  (Werbos:  1991),  (Won,  Cho:  2003), etc.  Generally  speaking,  “a  neural 
network  conducts  an  analysis  of  the  information  and  associates  a  probability  estimate  that 
the  data  matches  the  characteristics  it  has  been  trained  to  classify.”  (Sani  et  al:  2009). 

This  training  is  done  by  modifying  the  thresholds  of  the  system  so  that  the  neural  network 
returns  the  classifications  described  by  the  end  user.  Neural  networks  are  popular  because 
they  are  relatively  simple  to  design,  make  few  assumptions  regarding  the  underlying 
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distributions  of  elements  it  is  tuned  to  classify,  and  is  highly  adaptable  through  the 
training  process. 

Boolean  rules  make  use  of  Boolean  algebra  to  classify  elements  of  interest.  Two 
of  the  simplest  Boolean  rules  that  are  used  heavily  in  research  are  the  AND  (conjunctive) 
rule  and  the  OR  (disjunctive)  rule.  These  rules  can  be  combined  in  various  ways  to  create 
new  decision  or  voting  rules.  Popular  voting  rules  include  the  majority  vote  rule  and  the 
sensor  dominance  rule.  The  majority  vote  rule  considers  all  permutations  of  the 
application  of  the  AND  rule  to  the  classification  systems  and  then  applies  the  OR  rule 
between  each  combination.  The  sensor  dominance  rule  permits  a  single  classification 
system  to  dominate  the  decision  for  the  fused  classification  systems  ( Schubert :  2005). 
Some  novel  optimization  approaches  for  receiver  operating  characteristic  (ROC)  curves 
have  been  proposed  using  a  set  of  Boolean  functions.  The  ROC  curve  is  a  graphical  tool 
for  depicting  true  positive  rate  along  the  vertical  axis  (hit  rate)  as  compared  to  false 
positive  rate  along  the  horizontal  axis  (false  alarm  rate).  This  BCall  approach  developed 
by  (W.  Khreich  et  al:  2012).  Applies  a  set  of  a  10  Boolean  rules  to  two  ROC  curves  and 
chooses  the  rule  that  optimizes  performance  at  each  point  on  the  curve.  Most  Boolean 
rules  that  are  used  in  the  fusion  of  classification  systems  must  have  their  performances 
determined  after  the  actual  fusion  process.  The  work  of  ( Oxley  M.E.,  Bauer,  K.W.:2003) 
and  earlier  papers  began  the  notion  of  being  able  to  describe  the  performance  of  fused 
classification  systems  prior  to  any  sort  of  formal  testing.  This  was  done  by  deriving  an 
expression  for  the  performance  of  the  combined  system  using  properties  of  the 
performances  of  the  individual  classification  systems.  Oxley  and  Bauer  originally 
developed  an  expression  for  the  logical  OR  rule  assuming  independence  between  the 
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individual  classification  systems.  The  work  of  ( Schubert :  2005 )  and  (, Schubert  et  al: 

2004 )  has  extended  this  work  considerably,  developing  expression  for  the  logical  AND 
rule  and  developing  the  concepts  of  within  and  different  types  of  across  label  fusion.  This 
thesis  relies  on  the  theory  developed  in  these  fusion  methods.  In  these  works,  the 
performance  of  the  fused  classification  systems  are  expressed  through  the  use  of  the  ROC 
curve  using  the  assumption  of  a  two  class  label  set. 

Finally,  there  is  the  use  of  statistical  modeling  in  fusion  research.  Generally  this 
takes  the  form  of  some  type  of  generalized  linear  model  (GLM).  A  generalized  linear 
model  is  a  linear  regression  model  that  allows  the  response  variables  to  be  non-normally 
distributed.  However,  it  is  most  common  in  literature  to  assume  that  the  error  or  noise  in 
statistical  fusion  is  Gaussian  or  normally  distributed.  Though  it  is  uncommon  to  work 
with  non-Gaussian  statistical  models,  industrial  noise  often  shows  non-Gaussian 
characteristics  ( Niu ,  Zhu,  Gu,  Chu:  2009).  In  statistical  modeling,  the  goal  is  to 
approximate  the  distribution  of  those  elements  of  interest  generally  through  the  use  of 
least  squares  analysis  or  the  method  of  maximum  likelihood.  Some  advantages  of 
statistical  fusion  are  that  modem  software  packages  make  this  analysis  very  easy  to 
conduct.  A  statistical  model  can  be  rigid  in  the  fact  that  most  linear  regression  techniques 
require  that  error  terms  be  normal,  independent,  and  identically  distributed. 

2.2  Independence  and  its  Effects  on  Fusion 

Some  methods  described  above  make  no  assumptions  regarding  the  level  of 
dependence  between  classification  systems.  Most  neural  networks  do  not  make 
assumptions  regarding  the  level  of  dependence  between  elements  being  classified  or 
between  individual  networks  as  any  level  of  dependence  present  can  be  accounted  for  in 
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the  training  process.  Dependence  between  classification  systems  has  been  one  of  the 
hurdles  to  the  development  of  mathematical  expressions  for  the  performance  of 
classification  systems.  In  the  works  of  Oxley  and  Bauer,  it  was  assumed  that  the 
individual  classification  systems  to  be  fused  (and  therefore  their  ROC  curves)  are 
statistically  independent.  This  allows  for  the  Boolean  fusion  rules  to  be  defined.  In 
probability  theory,  if  two  events  are  independent,  then  the  probability  of  both  events 
occurring  simultaneously  is  equal  to  the  product  of  the  probabilities  of  the  individual 
events  occurring,  that  is,  the  occurrence  of  one  or  a  set  of  events  does  not  affect  the 
outcome  or  occurrence  of  another  event  or  set  of  events.  If  the  two  events  are  not 
independent,  then  calculating  the  probability  of  both  events  occurring  simultaneously 
may  either  not  be  tractable  or  it  may  be  very  difficult.  The  Boolean  AND  rule  for  fusing 
individual  classification  systems  takes  the  form  of  the  independent  AND  rule  in 
probability.  Analogously,  the  Boolean  OR  rule  takes  the  form  of  the  independent  OR  rule 
in  probability;  that  is,  the  sum  of  the  probability  of  events  A  and  B  minus  the  product  of 
the  probability  of  events  A  and  B. 

When  fusing  ROC  curves  using  label  based  fusion  as  presented  in  {Schubert: 
2005),  it  is  further  required  that  the  false  positive  values  (the  probability  of  false  alarm 
rate)  of  the  fused  classification  system  be  conditionally  independent  with  respect  to  the 
non-target  partition  of  the  event  set.  Conditional  independence  of  events  is  a  stronger 
assumption  than  simply  independence  of  events.  As  defined  in  ( Dawid :  1979),  two 
random  variables  X  and  Y  are  conditionally  independent  with  respect  to  a  third  random 
variable,  Z,  if  the  probability  of  X  and  Y  given  Z  is  the  product  of  the  probability  of  X 
given  Z  and  the  probability  of  Y  given  Z.  More  importantly,  this  should  imply  that  the 
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probability  of  X  given  Y  and  Z  is  equivalent  to  the  probability  of  X  given  Z.  That  is  to 
say,  any  information  about  the  random  variable  Y  is  superfluous  and  has  no  impact  on  the 
probability  of  X  given  Z.  Dawid  cautions  that  the  use  of  improper  distributions /random 
variables  can  lead  to  erroneous  results.  It  may  be  possible  in  this  instance  to  factor  the 
conditional  probability  of  X  and  Y  given  Z  into  the  product  of  individual  conditional 
probabilities,  but  the  probability  of  X  given  Y  and  Z  is  no  longer  equal  to  the  probability 
of  X  given  Z.  This  is  known  as  the  marginalization  paradox.  This  is  important  to  discuss 
here  as  the  research  of  this  thesis  along  with  the  work  of  Schubert,  Oxley,  and  Bauer 
requires  the  use  of  proper  random  variables.  If  this  is  not  the  case,  this  can  lead  to 
erroneous  results. 

The  notions  of  independence  between  label-fused  ROC  curves  were  eased  in  the 
work  of  ( Schubert :  2005 )  through  the  derivation  of  an  expression  for  the  correlation.  This 
is  a  unique  approach  in  that  the  level  of  dependence  between  classification  systems  can 
be  calculated  through  the  use  of  formulas  involving  only  the  ROC  curves.  Using  this 
concept,  functions  were  derived  that  accounted  for  a  fixed  level  of  correlation  and  hence 
independent  label- fused  ROC  curves  were  those  functions  where  the  correlation 
coefficient,  p,  is  zero. 

Finally,  there  has  been  some  research  into  the  effects  of  correlation  on  the 
performance  of  classification  systems.  In  (Petrakos,  Kaanelopoulous,  Benediktsson, 
Pesaresi:  2000 )  the  researchers  investigated  the  effects  of  correlation  on  fusion  of 
classification  systems  using  satellite  imagery  data.  They  did  this  by  assessing  the  measure 
of  agreement  between  different  classification  methods.  In  the  work  of  (Won,  Cho:  2003 ) 
the  researchers  selected  ideal  features  from  DNA  microarray  data  that  were  negatively 
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correlated  in  an  attempt  to  boost  the  performance  of  classification  methods.  This 
demonstrates  that  the  presence  of  dependence  can  actually  be  used  to  enhance  the 
performance  of  fused  classification  systems. 

2.3  Importance  of  Deriving  a  Method  for  Quantifying  Bias  in  Label  Fusion 

Classically  speaking,  the  term  bias  in  regards  to  classification  theory  generally 
implies  that  the  data  being  classified  is  unequally  weighted  or  that  a  subset  of  classes  is 
more  heavily  favored.  A  good  example  of  this  comes  from  ( Abiantun  and  Savvides: 
2009),  where  the  researchers  refine  the  Adaboost  algorithm  to  offset  the  bias  inherently 
built  into  facial  feature  classification  systems.  Up  to  this  point,  most,  if  not  all,  research 
into  the  effects  of  imbalanced  and  biased  classification  systems  has  been  done  with 
respect  to  feature  fusion.  Given  the  derivation  of  a  mathematical  expression  for 
quantifying  both  the  performance  and  correlation  of  classification  systems  as  provided  by 
{Schubert:  2005)  and  ( Schubert ,  Oxley,  and  Bauer:  2005),  it  has  become  possible  to 
characterize  the  inherent  bias  that  exists  in  label  fusion.  The  purpose  of  this  thesis  is  to 
form  a  method  and  an  algorithm  to  both  quantify  and  adjust  for  the  bias  that  exists 
between  different  types  of  label-fused  classification  systems  with  respect  to  event  set 
partitions.  This  is  important  for  the  following  reasons.  To  date,  there  has  been  almost  no 
investigation  into  the  effect  of  event  set  partitions  on  the  performance  of  the  fused 
system.  Secondly,  any  bias  that  does  exist  between  different  label  fusion  methods  could 
possibly  be  used  as  tools  to  tweak  the  performance  of  label-fused  systems. 
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Ill  Methods  and  Theoretical  Development 


The  mathematical  theory  of  classification  systems  and  Boolean  fusion  rules  will 
be  developed  here.  This  will  be  achieved  by  developing  the  notion  of  conditional 
probability  to  describe  the  performance  of  a  classification  system,  the  receiver  operating 
characteristic  (ROC)  curve  to  evaluate  classification  system  performance,  and  defining 
the  across  and  within  label  fusion  rules.  Further  detail  for  much  of  these  methods  and 
theoretical  development  can  be  found  in  {Schubert,  et  al:  2005)  and  (Schubert:  2005). 

3. 1  Classification  Systems 

3.1.1  Single  Classification  System 

Define  T  to  be  a  population  set  of  outcomes,  i.e.,  the  overarching  or  underlying 
event  set.  Let  G  be  a  c-algebra  of  subsets  of  T.  Then,  (T,  G )  defines  a  measurable  space 
{Schubert:  2005).  Define  Pp  to  be  a  probability  measure  defined  on  G.  This  implies  that 
(T,  G,  Pp)  defines  a  probability  measure  space.  Let  s  be  a  sensor  that  maps  outcomes  from 
T  to  a  new  data  set,  A.  Let  D  be  a  a-algebra  on  A,  implying  that  (A,  D)  defines  a 
measurable  space.  Furthermore,  define  Pa  to  be  a  probability  measure  defined  on  D; 
therefore  (A,  D,  PA)  defines  a  probability  measure  space.  Some  examples  of  data  sets  may 
include  different  segments  of  the  electromagnetic  spectrum,  search  engine  keywords,  or 
images.  Sometimes  this  data  is  too  nebulous  to  make  an  accurate  decision,  so  another 
mapping p  (a  processor)  is  defined  on  A  that  can  be  used  to  produce  an  object / which  is 
called  a  feature.  Let  O  be  a  feature  set  and  define  F  to  be  a  a-algebra  of  subsets  of  ®. 

This  makes  (®,  F)  a  measureable  space.  Letting  P<j,  be  a  probability  measure  defined  on  F 
defines  the  probability  measure  space  (®,  F,  P<d).  In  most  circumstances,  including  this 
thesis,  the  feature / is  a  vector  of  real  numbers,  though  it  need  not  be.  Let  0  be  a  set  of 
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parameters.  For  each  0  6  0,  let  ae  be  a  classification  system  mapping  that  takes  elements 
of  ®  into  A,  the  label  set.  Defining  L  to  be  a  a-algebra  on  A  makes  (A,  L)  a  measureable 
space.  Letting  PA  be  a  probability  measure  defines  a  probability  measure  space  (A,  L, 
Pa).  The  label  may  take  any  number  of  forms,  but  in  this  thesis,  elements  from  A  will 
take  the  form  {non-target,  target}  =  {n,  t},  depending  on  context.  The  composition  of 
these  mappings  creates  the  single  classification  system,  defined  in  this  context  to  be  Ae. 
That  is,  for  every  0  6  0: 


Ae  =  ae  o  p  o  s 


r 


Ae 


A 


(3.1) 


3.1.2  Multiple  Classification  Systems 

It  is  possible  to  combine  two  or  more  classification  systems  together.  In  this 
thesis,  only  the  fusion  of  two  classification  systems  is  considered.  Further,  the  systems  to 
be  combined  are  fused  together  using  label  fusion.  Label  fusion  may  be  loosely  defined 
as  the  joint  system  decision  based  on  processing  the  labels  (the  decisions)  given  to 
elements  in  the  event  set  by  individual  classification  systems.  Consider  two  classification 
systems:  A@  and  B^.  Let  A@  be  the  system  defined  previously.  Let  system  B*  be  defined  as 
the  classification  system  defined  by  the  composition  mapping: 


B„  =  bn  o  p2  o  s2  (3.2) 

Classifier  b*  maps  elements  from  the  feature  set  associated  with  system  B*  into  its  label 
set.  Note  that  the  sensor,  S2,  and  processor,  p2,  are  different  from  those  processors  and 
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sensors  defined  for  classification  system  Aq.  Now,  consider  that  these  two  classifications 


systems  observe  the  same  element,  x,  in  the  event  set,  f.  That  is,  for  x  6  f: 


Ae  =  aeop  i°  SiO) 


(3.3) 


Bn  =  K0P2°  s20) 


The  labels  produced  by  these  two  compositions  are  then  fused  to  create  a  system  label  as 


generated  by  both  classification  systems,  hence  label  fusion.  The  following  schematic 


outlines  this  process  of  label  based  fusion. 


Figure  3-1  -  Mapping  of  element  from  the  population  of  event  sets,  r,  into  the  fused  label  set.  A'. 


It  is  possible  to  develop  other  system  mappings.  Say  for  example  that  the  two  data  sets 
from  above  are  simultaneously  mapped  to  the  same  feature  set. 
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Figure  3-2  -  Alternate  mapping  of  elements  from  f  into  A'.  Note  here  that  px  and  p2  map  into  the  same  feature  set. 


O'. 


The  act  of  fusing  information  at  an  earlier  stage  (fusing  into  the  same  feature  set) 
may  change  the  way  that  the  classification  systems  label  events.  This  is  largely  a  question 
of  design  on  the  part  of  the  experimenter  and  the  environment  in  which  the  classification 
system  is  to  be  applied.  Other  types  of  mapping  exist,  but  this  thesis  is  concerned  with 
those  systems  A@  and  that  map  into  fused  label  sets. 

3.1.3  Properties  of  Classification  System  Mappings 
Before  developing  a  formulaic  approach  to  analyze  the  data  returned  by  the 
classification  system,  a  few  properties  must  be  outlined  here. 

Definition  3.1  (Pre-image)  Let  X and  Y  be  nonempty  sets.  Let  fbe  a  mapping  that  takes 
an  element  x  EX  into  Y.  Given  some  subset  Y  a  Y,  the  pre-image  off  is  defined  to  be  the 


subset  in  X  such  that 


f\Y]  =  { xEX:f(x)EY } 
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Hence,  the  pre-image  of  a  subset  Y  oY  are  those  elements  in  X  that  are  mapped  by  f 
into  Y. 

In  most  mathematical  texts,  the  pre-image  is  generally  considered  synonymous 
with  the  inverse  image.  Considering  that  the  mapping/ described  above  need  not  be 
invertible,  the  natural  symbol  (f\)  as  used  by  ( Schubert ,  C.,  Oxley,  M.  E.,  Bauer,  K:  2005) 
will  be  used  here.  The  pre-image  for  a  classification  system  allows  a  way  to  map 
backwards  from  the  label  set  into  some  subset  or  element  from  the  population  event  set, 

T.  That  is,  the  image  and  pre-image  for  Ae  can  be  written  respectively  as  follows. 

Aq  =  CLq  O  P  O  s 

Aq  =  a.Q  o  p*  o  s*  ft-4) 

The  performance  associated  with  a  certain  classification  system  is  assessed  using 
probability  theory.  Therefore,  it  will  be  necessary  to  develop  the  notion  of  a  measurable 
space  and  probability  measure  space  so  that  this  may  be  done. 

Definition  3.2  (Measurable  Mapping)  Let  2  be  a  o-algebra  of  subsets  of  setX.  Let  W  be  a 
o-algebra  of  subsets  of  set  Y.  This  implies  that  (2,  X)  and  (W,  Y)  are  measurable  spaces. 

A  mapping  f  is  measurable  if for  every  subset  ip  GW 

f\w]  =  {xi>ew-.f\xi>)es} 

That  is,  the  pre-image  f  (\|/)  G  S. 

Using  this  definition,  the  composition  mapping  described  by  Af  is  measurable.  Thus, 

Af:  L-oD.  That  is,  the  mapping  takes  the  subsets  from  the  o-algebra  associated  with  the 
label  set,  A,  into  the  o-algebra  associated  with  the  data  set,  A.  It  is  implied  that  the 
composition  of  measurable  mappings  must  take  the  subsets  from  the  o-algebra  associated 
with  A  into  the  o-algebra  associated  with  the  feature  set,  ®,  before  mapping  back  into  the 
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c-algebra  of  the  data  set.  Furthermore,  if  it  is  assumed  that  the  composition  is 
measurable,  a  random  variable  may  be  defined  for  the  system  in  question.  Consider  the 
measurable  mapping  defined  by  G  =  p(s).  This  mapping  is  a  called  a  random  element  of 
the  feature  set,  ®,  and  is  called  a  random  variable  when  (®,  F)  =  (M,  F(M))  ( Schubert : 
2005). 

Definition  3.3  -  (Measurable  Mapping)  Let  (T,  G)  and  (0,  F)  be  measurable  spaces.  M 
is  called  a  random  mapping  if  M:  r  -t>  0  is  a  T-0  measurable  mapping. 

Definition  3.4  -  (Induced  Probability  Distribution)  Let  Pp  be  a  probability  measure 
defined  on  G  such  that  (P,  G,  Pp)  defines  a  probability  measure  space.  Further,  suppose 
that  (0,  F)  defines  a  measurable  space.  Let  M:  r  -t>  0  be  a  random  mapping.  Define  the 
set  function  [Pp  oM]^  on  F 

[ProM)YXf)  =  Pr(M\n) 

for  every  f  EF.  Thus,  P&  =  Pp  oM'  defines  a  probability  measure  on  (0,  F)  known  as  the 
induced  probability/distribution  measure  of  M  and  (0,  F,  P$)  is  a  probability  measure 
space. 

The  concept  and  formulation  of  measurable  mappings  is  necessary  in  order  to 
analyze  classification  systems.  Further  information  and  technical  development  may  be 
found  in  ( Schubert :  2005).  Measurable  mappings  and  more  importantly,  probability 
distributions  and  measure  spaces,  allows  a  way  to  assign  probability  outcomes  that  can  be 
mapped  back  to  the  original  event  sets.  This  is  clearly  important  given  that  under  most 
circumstances,  the  original  event  is  unknown.  In  this  thesis,  these  concepts  will  be  used 
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to  develop  a  way  to  quantify  the  difference  in  performance  between  within  and  across 
label  fusion. 

3.2  Performance  of  Classification  Systems 

After  the  classification  system  of  interest  has  assigned  labels  to  the  elements  in 
the  population  event  set,  it  is  still  unknown  how  well  the  system  has  performed.  It  is 
highly  unlikely  that  the  classification  system  has  worked  exactly  as  intended,  i.e.,  made 
perfect  classifications.  Therefore,  there  must  be  a  way  to  quantify  the  errors  and 
successes. 

The  receiver  operator  characteristic  curve  allows  a  way  to  qualitatively  and 
quantitatively  assess  classification  system  performance.  The  ROC  curve  is  useful  as  it 
graphically  depicts  the  difference  between  false  alarm  rate  and  true  hit  rate  for  each 
parameter  threshold  of  the  classification  system.  The  classification  system  produces  a 
true  positive  (TP)  when  it  labels  an  element  from  the  target  population  event  set  to  which 
it  is  tuned  as  a  “target.”  The  classification  system  produces  a  false  positive  (FP)  when  it 
labels  an  element  from  the  non-target  population  event  set  as  “target.”  A  ROC  curve  is 
created  by  graphing  FP  rates  along  the  horizontal  (X)  axis  and  TP  rates  along  the  vertical 
(Y)  axis.  The  classification  system  approximates  the  true  ROC  curve  using  empirical  data 
produced  during  the  composition  of  mappings. 

There  are  four  possible  probability  outcomes  given  that  an  event  can  receive  a 
target  or  non-target  label  and  the  event  set  is  partitioned  into  target  and  non-target 
subsets.  This  is  valid  given  that  a  probability  measure  space  has  been  defined.  Consider 
system  Bn.  Note  that  T  is  composed  of  all  subsets  of  targets  and  non-targets.  For  the  sake 
of  simplicity,  assume  that  T  is  divided  into  two  populations.  Denote  the  event  set 
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composed  of  targets  system  B*  is  tuned  to  classify  as  rt.  Denote  the  event  set  composed 
of  non-targets  system  is  tuned  to  identify  as  rn.  Let  Ptp(B^)  denote  the  probability  that 
Bjt  correctly  labels  an  element  t  e  rt  as  a  “target.”  This  is  the  definition  of  a  true  positive 
classification  by  system  Brt  as  the  system  correctly  labeled  a  target  with  the  target  label. 
Mathematically,  this  can  be  modeled  using  conditional  probability: 


Ptp  (Bn)  —  —  t  •  tB  £ft} 


p (Bni.it)  n  rt) 

p(Bt) 


(3.5) 


Let  Pfp(B^)  denote  the  probability  that  B*  incorrectly  labels  an  element  n  G  rn  as  a 
“target.”  This  is  a  measure  of  a  false  positive  classification  by  system  B„  as  the  system 
falsely  labeled  the  non-target  element  with  a  target  label.  This  too  may  be  modeled  using 
conditional  probability 


Pfp (BJ  =  P{B„(nB )  =  t  :nB  ETt}  = 


P(Pn) 


(3.6) 


Next,  let  Ptn(B^)  denote  the  probability  that  B*  correctly  labels  an  element  n  G  Tn 
as  a  “non-target.”  This  is  the  definition  of  a  true  negative  classification  by  system  B^  as 
the  system  correctly  labeled  a  non- target  with  the  non- target  label.  Mathematically 
speaking: 


Ptn(Bu )  =  P{Bn(nB )  =  n:  nB  E  rn}  = 


p(g^(Lra)nrra) 

P(Pn) 


(3.7) 


Lastly,  let  Pfn(B^)  denote  the  probability  that  B*  incorrectly  labels  an  element  t  G 
Tt  as  a  “non-target.”  Then  Pfn(B^)  is  the  probability  that  B*  produced  a  non-target  label 
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for  an  element  that  was,  in  truth,  a  target.  This  is  more  commonly  known  as  a  false 
negative  designation  and  is  described  mathematically  as  follows: 


Pfn(Biz)  —  —  n  •  tB  E  rn}  — 


p(^(Lra)nrt) 

P(rt) 


(3.7) 


These  definitions  lead  to  two  important  properties  involved  with  these  probability 
statements. 


r  mi  r  fP._p«ynr«)  P«(ynr,) 

=  P(B*qt)  n  rt)  +  p(g*(Lra)  n  rt) 

W) 

p((B^(Lt)uBX))nft) 


nn) 

pin ) 


pin) 


=  i 


(3.8) 


„  rp  ,  ,  „  rp  .  _  p  W(Lt)  n  rn)  p{B%(Ln)  n  rn)  (3.9) 

Pfp(bj + P™(B  J  -  wo  +  wo 

P(gW  n  rw)  +  pQWn)  n  rw) 

PiPn) 

p((5W)usWn))nrn) 


Wn) 


P(Pn) 


=  1 


PiPn) 

Note  that  for  any  of  the  above  probabilities,  all  are  dependent  upon  the  parameters 
of  the  classification  system.  A  single  probability  value  is  associated  with  a  specific 
parameter.  That  is  to  say,  for  each  parameter  combination,  a  TP,  FP,  TN,  and  FN  is 
associated  with  the  given  combination.  These  probabilities  change  as  the  parameter 
values  change.  Given  that  the  goal  is  to  produce  a  ROC  curve  for  the  classification 
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system  B*,  the  focus  will  be  on  the  TP  and  FP  values.  Define  n  to  be  the  parameter  set  for 
system  then  the  following  set  of  ordered  triples  defines  the  trajectory  of  B. 

T-b  =  {(ir,  PFP(Bn),  PTP(Bn)^  :  7T  6  77  } 

Projecting  the  trajectory  onto  the  FP  and  TP  components  yields  those  points  which  define 
the  frontier  (denoted  by  7)  of  the  ROC  curve. 

=  {(^fpC^tX^TpC^jt))  :  Tt  G  77  j 

Assuming  that  the  set  n  is  homeomorphic  to  R,  then  the  trajectory  corresponds  to 
a  curve  in  M3  and  the  frontier  corresponds  to  a  projection  onto  R2  (Schubert:  2005).  This 
frontier  is  the  ROC  curve  for  system  B*.  There  may  be  other  points  that  empirically  exist 
on  the  curve,  but  the  definition  of  a  ROC  curve  requires  that  the  ROC  function  be  non¬ 
decreasing.  Considering  that  both  PFp(B^)  and  Pjp(Bn)  G  [0,1],  7  b  is  a  projection  onto 
[0,1]  x  [0,1],  the  unit  square.  In  this  instance  the  parameter  set  is  one-dimensional; 
therefore  7 s  is  composed  of  a  single  curve  that  is  projected  onto  E  .  The  same  is  true  for 
classification  system  AH.  Next,  the  ROC  curve  and  ROC  function  is  formally  defined. 
Definition  3.5  (ROC  curve)  Assume  that  B  =  (Bre:  n  G  77  }  defines  the  classification 
family  of  interest.  Let  p  G  [0,  1]  correspond  to  the  value  of  the  false  positive.  Similarly,  let 
q  G  [0,  1]  correspond  to  the  value  of  the  true  positive.  The  ROC  curve  for  “B  is  defined  as 
T-b  =  {(p,  q)  •  p  G  [0, 1]  and  q  =  max{PTP(Bn )  :  7r  G  77  and  PfP  (.Bn)  —  P  )} 
and  the  corresponding  ROC  function  may  be  defined  as 

^®(p)  =  niax{PTP(Bn)  :  7r  G  77  and  PFp(Bn)  <  p  } 
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3.3  Across  Label  Fusion  and  Within  Label  Fusion 


Now  that  a  method  is  in  place  to  create  the  ROC  curve  for  individual 
classification  systems,  it  is  important  to  consider  how  to  fuse  these  systems  in  an  attempt 
to  increase  predictive  power.  In  order  to  use  the  performance  of  the  individual 
classification  systems  to  quantify  the  performance  of  the  fused  system,  there  must  be 
some  accounting  for  the  different  target  and  non-target  sets  that  the  classification  systems 
are  tuned  to  label.  Some  formal  definitions  regarding  how  the  classification  systems 
partition  the  event  set  has  been  developed  and  will  be  briefly  touched  upon  here. 
Definition  3.6  (Finite  Partition)  Let  S  be  a  non-empty  set. 

1)  If  Am  D  An  =  0  Vm  =  1,  ...,  M and  n  =  1,  ... ,  N;  mf  n;  M,  N  <  oo  (Pairwise 
disjoint) 

2)  U ?=1At=S 

Let  {Aj,  ...,  Ak,  ...}  be  a  countable  collection  of  subsets  ofS.  Then  {A /,  ...,  Ak,  ...} 

forms  a  finite  partition  of  S. 

Definition  3.7  (T rue  Partition)  Assume  the  following 

1)  (T,G)  is  a  measurable  space 

2)  A  =  {a  I,  A  7,  ....  Xm}  is  a  finite  label  set.  A  is  the  power  set  of  A  such  that  (A,  A)  is  a 
measurable  space. 

3)  t :  T  -o  A  is  a  measurable  mapping .  The  domain  of  t  is  T  and  the  range  of  t  is  A. 

Then  t  defines  a  truth  mapping.  The  collection  of  pre-images  (T ),  T ),  ...,  F^J  defined 

by  Tn  =  tq({Xn})  EG  Vn  =  1,  ...,  N forms  a  partition  of F  that  is  the  true  partition  with 

respect  to  the  truth  mapping,  t.  (Schubert:  2005) 
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Definition  3.8  (Within-Fusion  Rule)  Assume  the  following 

1)  (r,  G)  is  a  measurable  space. 

2)  A  =  {Xj,  A2,  ...,  Am}  is  a  finite  label  set.  A  is  the  power  set  of  A  such  that  (A,  A)  is  a 
measurable  space. 

V  Ga  =  {rXl,  rXz, ,  TXm}  is  the  truth  partition  of  T  with  respect  to  A. 

If  the  classification  systems  Si,  S2,  ...,  Sn  :  T  -0  A  are  measurable  mappings 
designed  to  map  I\M  —t>  Am  for  each  m  =  1,  ...,  M,  then  the  fusion  rule  r  that 
combines  the  collection  of  classification  system  systems  yielding  the  new 
classification  system 

Vo  =  r(Bi>B2, 

is  said  to  be  a  within-fusion  rule.  ( Schubert :  2005) 

Definition  3.9  (Across-Fusion  Rule)  Assume  the  following 

1)  (F,  G)  is  a  measurable  space. 

2)  A  =  {A],  A2,  ...,  Am}  is  a  finite  label  set.  A  is  the  power  set  of  A  such  that  (A,  A)  is  a 
measurable  space. 

V  ga  =  {rXl,rX2, ...,rXM)  cc  is  the  truth  partition  ofF  with  respect  to  A. 

4)  A(0>,  A(l>,  ...,  A(N)  a  A  are  partitions  of  A. 

5)  For  each  n  =  0,  ....  N,  the  integer  M(n>  =card(A(n>)  <  M,  and  the  partition  A(n>  is 

congruent  to  the  label  set  A ^  ... ,  j 

6)  For  each  n  =  0,  ...,  N,  the  partition  G(n}  cz  G  is  the  true  partition  of  F  with  respect 
toA(n). 
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If  the  classification  systems  Si :  r  -»A(1\  Sj :  r  —t>A<2>,  Sn  :  T  —t>A<N>are 
designed  to  map  each  event  set  rw  E  G(n>  to  the  corresponding  w  EA(n>,  then  for 
every  n  =  1,  2,  N,  the  fusion  rule  r  that  combines  the  collection  of 
classification  systems  yielding  the  new  classification  system  system 

So  =  rfB^, 

is  said  to  be  an  across-fusion  rule.  {Schubert:  2005) 

For  the  sake  of  clarification,  a  within  label-fused  system  is  composed  of 
individual  classification  systems  which  are  both  tuned  to  the  same  target  and  non¬ 
target  partitions  of  the  event  set.  An  across  label-fused  system  is  composed  of 
individual  classification  systems  that  are  tuned  to  different  partitions  of  the  target  and 
non-target  event  set. 

3.3.1  Within  Label  Fusion 

Assume  there  exists  two  classification  systems,  Ae  and  B„,  which  are  to  be  fused 
under  within  label  fusion.  Further,  assume  that  classification  systems  A@  and  are  both 
designed  to  classify  a  single  target  type  and  a  single  non-target  type.  Thus,  the  label  set  in 
question  is  A  =  {t,  n}  where  t  denotes  “target”  and  n  denotes  “non-target.”  Furthermore, 
Ga  =  {Tt,  rn}  c  G  is  the  true  partition  of  Y  with  respect  to  A.  Hence,  system  Ao :  T  -t>  A 
where  AH  is  designed  to  map  Tt  G  Ga  to  the  corresponding  t  G  A  and  rn  G  Ga  to  the 
corresponding  n  G  A.  The  partitions  of  the  event  set  and  label  sets  of  systems  Ae  and  B„ 
are  the  same.  That  is,  for  any  target  element,  systems  A@  and  B*  will  label  the  element 
using  the  same  “t”  label. 


44 


3.3.2  Across  Label  Fusion 


There  are  three  ways  that  classification  systems  may  be  categorized  as  across 
label  fusion  in  this  thesis  as  previously  outlined  by  ( Schubert :  2005). 

1)  Case  I  -  Each  classification  system  labels  mutually  disjoint  targets. 

2)  Case  II  -  One  classification  system  labels  a  subset  of  targets  of  the  other  classification 
system. 

3)  Case  III  -  The  targets  of  the  two  classification  systems  overlap,  creating  a  subset  of 
targets  labeled  by  both  systems. 

Case  I  -  Each  classification  system  labels  mutually  disjoint  targets. 

Assume  classification  system  AH  is  designed  to  classify  target  type  1  (ti)  and 

classification  system  system  B„  is  designed  to  classify  target  type  2  (t2).  Thus,  the  label 

set  of  interest,  A  =  {t,  n}  where  t  =  U  t2  and  n  t2  =  0.  GA  =  (rtl,  rt2,  rn}  c  G  is  the 

true  partition  of  T  with  respect  to  A.  The  label  set  for  classification  system  A@  is  A(A)  = 

{ti,  ni}  and  the  label  set  for  classification  system  EC  is  A('B)  =  {t2,  n2}.  Note  that  ni 

denotes  the  complementary  non-target  type  outcome  that  is  composed  of  both  “n”  and 

“t2”  elements.  Similarly,  n2  denotes  the  complementary  non-target  outcome  that  is 

composed  of  both  “n”  and  “ti”  elements.  Hence,  Ae :  T  -»  A(A)  where  Ae  is  designed  to 

map  Tt  6  G(A)  to  the  corresponding  ti  G  A(A)  and  rniG  G(A)  to  the  corresponding  ni  E 

A(a).  Analogously,  B*  :  T  -o  A(®’  where  Bn  is  designed  to  map  Tt2G  G<s)  to  the 

corresponding  t2  G  A1®*  and  Tn2G  G1®*  to  the  corresponding  n2  G  A(®}. 

Case  II-  One  classification  system  labels  a  subset  of  targets  of  the  other  classification 
system. 
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Assume  classification  system  AH  is  designed  to  classify  ti  events  and  assume 
classification  system  B*  is  designed  to  classify  any  target  type  (t).  Thus,  the  label  set  of 
interest,  A  =  {t,  n}.  GA  =  (rtl,  rt2,  rn}c  G  is  the  true  partition  of  T  with  respect  to  A.  The 
label  set  for  classification  system  Ae  is  A(A)  =  {ti,  ni}  and  the  label  set  for  classification 
system  B*  is  A<B)  =  {t,  n}.  Note  that  ni  denotes  the  complementary  non-target  type 
outcome  that  is  composed  of  both  “n”  and  ‘V’  elements.  Hence,  Ao :  T  -t>  AiA|  where  A@ 
is  designed  to  map  rtie  G|A|  to  the  corresponding  ti  G  A(A)  and  rniG  G|A)  to  the 
corresponding  ni  G  A(A).  Analogously,  B*:  T  -t>  A($)  where  B*  is  designed  to  map  TtG  G('B) 
to  the  corresponding  t  G  A<B)  and  TnG  G(s)  to  the  corresponding  n  G  A<s). 

Case  III  -  The  targets  of  the  two  classification  systems  overlap,  creating  a  subset  of 
targets  labeled  by  both  systems. 

Assume  classification  system  A@  is  designed  to  classify  target  types  1  and  2  (ti 
and  t2).  Assume  classification  system  B*  is  designed  to  classify  target  types  2  and  3  (t2 
and  t3).  The  label  set  is  A  =  {t,  n}  where  t  =  t±  U  t2  U  t3  and  tx  n  t3  =  0.  GA  = 

{rti<  rt2  ,  rt3rn}  c  G  is  the  true  partition  of  T  with  respect  to  A.  The  label  set  for 
classification  system  AH  is  A(a)  =  {ti,  t2,  ni2}  and  the  label  set  for  classification  system  B* 
is  A1®*  =  {t2,  t3,  n23}.  Note  that  ni2  denotes  the  complementary  non-target  composed  of 
both  ‘V’  and  “n”  labels  for  system  Ae.  Similarly,  n23  denotes  the  complementary  non¬ 
target  type  for  system  B„  composed  of  both  “ti”  and  “n”  elements.  Hence,  Ae :  T  -i>  Ai  A) 
where  Ae  is  designed  to  map  {rtl,  rt2}  G  G(A)  to  the  corresponding  ti2  6  A(A)  and  {Tn,  Tt3} 

G  G(A)  to  the  corresponding  ni2  G  A(A).  Analogously,  Brt:  T  -i>  A(B)  where  B„  is  designed 
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to  map  {rt2,  rt3}  G  G(®’  to  the  corresponding  t23  G  A(®’  and  {rn,  rti}G  G(®’  to  the 
corresponding  n23  G  A1®'. 

3.4  Label  Fusion  Rules 

In  this  thesis,  the  focus  is  within  and  across  label-fusion  under  the  Boolean  AND 
and  Boolean  OR  rules. 

3.4.1  Boolean  AND  Label  Fusion  Rule 

The  AND  rule  is  a  binary  operation  that  is  defined  on  the  label  set,  A.  This 
operator  will  be  defined  using  the  logical  AND  symbol,  A.  It  is  defined  in  the  following 
table. 


A 

t 

n 

t 

t 

n 

n 

n 

n 

Table  3-1  -  Truth  table  of  label  outcomes  for  Boolean  AND  rule. 

Now,  consider  classification  systems  Ae  and  Bn.  The  Boolean  AND  label- fused 
classification  system  C“en^  may  be  defined  as  the  following. 

c(0Nto(*)  =  Ae(pO  A  Bn(x)  V  x  G  r 

That  is  to  say,  the  fused  classification  system  returns  a  target  label  (t)  only  when 
both  classification  systems  A0  and  Bn  label  the  same  element  in  question  as  being  a  target 
from  the  target  population  event  set.  The  fused  classification  system  returns  a  non-target 
label  (n)  when  either  A0  or  B„  returns  a  non-target  label  or  A0  and  B*  both  return  a  non¬ 
target  label.  The  AND  fusion  rule  is  sometimes  called  a  conservative  label/decision  rule 
because  it  requires  that  both  individual  classification  systems  label  the  element  as  a  target 
in  order  to  receive  a  combined  target  classification.  Note  that  under  the  fusion  rule  it  is 
not  known  which  system  returned  the  non-target  label  or  whether  both  systems  returned 
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the  non-target  label.  The  only  information  available  is  the  combined  decision  made  by  the 
fused  system.  If  knowledge  of  the  feature  set  is  available,  implying  that  the  true  partition 
of  the  event  sets  is  actually  known,  then  it  can  be  determined  which  system  or  systems 
returned  the  non-target  label. 

3.4.2  Boolean  OR  Label  Fusion  Rule 

The  Boolean  OR  rule  is  also  a  binary  operation  defined  on  A.  The  OR  operator 
will  be  defined  using  the  logical  OR  symbol,  V.  It  is  defined  in  the  following  truth  table: 


V 

t 

n 

t 

t 

t 

n 

t 

n 

Table  3-2  -  Truth  table  of  label  outcomes  for  Boolean  OR  rule. 


Consider  systems  A@  and  B„.  The  Boolean  OR  label-fused  classification  system  is  defined 
as  the  following 

c(6Mt)(*)  =  Ae(x)  V  Bn(x)  V  x  6  r 

Using  the  truth  table  as  a  reference,  notice  that  the  combined  OR  classification 
system  labels  an  element  x  G  T  as  a  target  if  either  one  or  both  classification  systems 
labels  the  element  as  a  target.  A  non-target  label  can  only  occur  when  both  individual 
classification  systems  label  the  element  in  question  as  a  non-target.  Similar  to  the  AND 
fusion  rule,  without  knowledge  of  the  true  partition  of  the  event  set,  it  is  not  known  which 
classification  system(s)  labeled  the  element  as  a  target. 

3.4.3  Within  AND/OR  Label  Fusion 

Now  that  the  Boolean  rules  are  in  place,  they  can  be  applied  to  within  label 
fusion.  Consider  first  the  within  AND  label  fusion  rule,  which  will  henceforth  be  referred 
to  using  the  shorthand  Aw.  Recall  that  in  within  label  fusion,  systems  AH  and  B^  classify 
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the  same  target  and  non-target  types.  Let  t  denote  the  targets  and  n  denote  the  non-targets 
that  A@  and  B,T  are  tuned  to  classify.  Consider  the  following  table  that  outlines  the  four 


possible  outcomes  for  systems  AH  and  B,T. 


Truth 

Ae 

B,t 

t 

n 

t 

n 

t 

TP 

FP 

TP 

FP 

n 

FN 

TN 

FN 

TN 

Table  3-3  -  Within  AND  label  fusion  for  individual  classification  systems  Ae  and  Bn. 


The  corresponding  probabilities  for  these  outcomes  as  determined  by  label  fusion  are 


outlined  below. 


Aw 

Tt  (True  target  partition) 

Tn  (True  non-target 
partition) 

AV«)  n 

TP 

FP 

AV{t})  n  B\({n}) 

FN 

TN 

AVW)  n  B\({t}) 

FN 

TN 

AVW)  n  B \({n}) 

FN 

TN 

Table  3-4  -  Within  AND  label  fusion  for  fused  classification  system  with  respect  to  partitions  of  the  event  set. 


As  can  be  seen,  the  errors  made  by  the  individual  classification  systems  add  to  the 
probability  of  a  FN  outcome  for  the  fused  classification  system.  However,  the  fused 
system  determines  true  non-targets  under  the  combined  Aw  label  fusion  rule  well  due  to 
the  large  number  of  partitions  for  a  correct  TN  outcome. 

Next,  consider  the  within  OR  label  fusion  rule  which  will  be  referred  to  using  the 
shorthand  notation  Vw  Table  3.3  also  applies  to  the  within  OR  label  fusion  rule,  but  the 
outcomes  with  respect  to  the  different  target  partitions  is  different  under  the  OR  rule. 


vw 

Tt  (True  target  partition) 

Tn  (True  non-target 
partition) 

AV{t})  u 

TP 

FP 

AV{t})  u  B\({n}) 

TP 

FP 

A>{n})  U  B\({t}) 

TP 

FP 

AVW)  u  BVW) 

FN 

TN 
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Table  3-5  -  Within  OR  label  fusion  for  combined  classification  system. 


Notice  that  under  the  within  OR  rule  that  the  fused  system  excels  at  correctly 
classifying  targets  in  the  environment.  However,  if  either  one  or  both  classification 
systems  incorrectly  labels  a  non-target  with  a  target  label,  this  significantly  adds  to  the  FP 
probability. 

3.4.4  Across  AND/OR  Label  Fusion 

Recall  that  under  across  label  fusion,  the  target  and  non-target  event  sets  are 
partitioned  in  three  different  ways.  For  case  I,  classification  system  Ae  is  tuned  to  ti  and 
the  complementary  non-target  ni.  Classification  system  EC  is  tuned  to  t2  and  the 
complementary  non-target  set  composed  of  ti  and  n,  that  is,  n2.  Symbolically,  the  across  I 
AND  rule  will  be  defined  as  A1 .  The  following  truth  table  outlines  the  target  and  non¬ 
target  designation  of  the  A1  combined  classification  system. 


A7 

Ae 

EC 

ti 

ni  =  {n,t2} 

t2 

t 

n 

n2=  {n,ti} 

n 

n 

Table  3-6  -  Across  I  AND  truth  table  for  fused  system. 

In  terms  of  the  four  probability  outcomes  and  their  associated  probability 
statements 


Truth  under  A7rule 

Ae 

EC 

ti 

P 

II 

'P 

to*" 

t2 

n2=  {n,ti} 

ti 

TP 

FN 

FP 

TN 

k 

FP 

TN 

TP 

FN 

n 

FP 

TN 

FP 

TN 

Table  3-7  -  Truth  table  for  individual  classification  systems  under  across  I  AND  label  fusion. 
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A7 

rt  (True  target  type 

I  partition) 

rt2  (True  target 
type  II  partition) 

Tn  (True  non-target 
partition) 

A'e({ti})  D  B\({t2}) 

TP 

TP 

FP 

AV{ti})nBV{n2}) 

FN 

FN 

TN 

AV{ni})nB\({t2}) 

FN 

FN 

TN 

AV{ni})n 

BV{n2}) 

FN 

FN 

TN 

Table  3-8  -  Fused  across  I  AND  classification  system  outcomes  with  respect  to  true  partitions  of  the  event  set. 


There  is  something  peculiar  about  the  individual  classification  systems  under  the 
A7  fusion  rule.  The  system  will  produce  a  FP  if  Ae  gives  the  t2  element  a  ti  label  or  it  can 
produce  a  TN  if  it  labels  the  t2  element  as  a  non-target.  Similarly,  classification  system  B,T 
has  a  similar  issue  when  classifying  elements  from  the  ti  partition.  If  B*  labels  the  ti 
event  with  a  t2  label,  then  the  classification  system  produces  a  FP,  but  if  it  labels  the  same 
ti  element  with  an  n  label,  it  produces  a  TN  classification.  Referring  to  the  table  above, 
this  produces  some  interesting  results  for  the  fused  system.  It  appears  that  if  Ae  correctly 
identifies  a  type  I  target  with  the  ti  label  and  B^  labels  the  same  element  as  belonging  to 
n2,  then  the  system  should  produce  a  true  positive  event.  The  reason  this  does  not  occur  is 
that  unless  the  true  partition  of  the  event  sets  is  known,  it  is  not  known  whether  the 
element  that  system  Bn  labeled  was  a  true  non-target  or  a  ti. 

Consider  the  across  I  OR  label  fusion  rule,  which  is  denoted  V7.  Note  that  table 
3.7  remains  true  for  the  V7  combined  classification  system.  Consider  the  labels  with 
respect  to  the  true  partitions  of  the  event  set  and  label  sets: 


V7 

Ae 

B* 

ti 

ni  =  {n,t2} 

h 

t 

t 

n2=  {n,ti } 

t 

n 
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Table  3-9  -  Truth  table  for  across  I  OR  fused  classification  system. 


v7 

rt  (True  target  type 

I  partition) 

rt2  (True  target 
type  II  partition) 

Tn  (True  non-target 
partition) 

AV{ti})  U  B\({t2}) 

TP 

TP 

FP 

AV{ti})UB\({n2}) 

TP 

TP 

FP 

AVM)  U  B\({t2}) 

TP 

TP 

FP 

AV{ni})U 

BV{n2}) 

FN 

FN 

TN 

Table  3-10  -  Across  I  OR  fused  classification  system  outcomes  with  respect  to  the  true  partitions  of  the  event  set. 


There  are  a  few  events  of  interest  in  table  3.10.  First,  consider  the  instance  where 
Ae  labels  the  element  “ti”  and  labels  the  element  as  “n2”  =  {ti,  n}  and  the  combined 
classification  system  returns  a  TP  with  respect  to  I\2 .  Both  individual  classification 
systems  in  this  case  are  wrong,  but  considering  that  A0  labeled  the  element  as  a  target 
(even  though  it  is  the  wrong  target  label),  the  combined  system  is  correct  in  labeling  the 
element  a  target.  The  other  unusual  occurrence  happens  when  A0  labels  the  element  with 
an  ni  label  and  B*  labels  the  same  element  with  a  t2  label.  With  respect  to  the  partition 
rt  ,  both  individual  systems  were  incorrect,  but  because  Brt  labeled  the  element  with  a 
target  label,  the  combined  classification  system  correctly  labeled  the  element  from  rtlas  a 
target. 


The  following  is  a  brief  treatment  of  the  outcomes  that  one  would  expect  from 
fused  across  II  and  across  III  fused  classification  systems.  Note  that  for  across  case  2 
(A"\V"),  the  target  partition  that  A0  is  tuned  to  classify  is  a  subset  of  the  target  partition 
that  classification  system  B„  is  designed  to  classify.  Recall,  also,  that  for  across  case  3 
(A'f,\v"')  that  A0  is  designed  to  classify  targets  ti  and  t2  and  classification  system  Brc  is 
designed  to  classify  targets  t2  and  t3,  where  t2  forms  a  subset  for  both  classification 
systems. 
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A11 

Ae 

B* 

ti 

ni  =  {n,t2} 

t=  {ti,  t2} 

t 

n 

n 

n 

n 

Table  3-11  -  Across  II  AND  combined  classification  system  truth  table. 


v" 

Ae 

B* 

ti 

ni  =  {n,t2} 

t  =  {ti,  t2} 

t 

t 

n 

t 

n 

Table  3-12  -Across  II  OR  combined  classification  system  truth  table. 


A11 

rt  (True  target  type 

I  partition) 

rt2  (True  target 
type  II  partition) 

Tn  (True  non- target 
partition) 

A"e({ti})  n  B\({t}) 

TP 

TP 

FP 

AV{ti})DBV{n}) 

FN 

FN 

TN 

AV{m})nBV{t}) 

FN 

FN 

TN 

AV{ni})  n  B\({n}) 

FN 

FN 

TN 

Table  3-13  -  Across  II  AND  fused  classification  system  outcomes  with  respect  to  the  true  partitions  of  the  event  set. 


V" 

rt  (True  target  type 

I  partition) 

rt2  (True  target 
type  II  partition) 

rn  (True  non-target 
partition) 

AV{ti})UBV{t}) 

TP 

TP 

FP 

A>{ti})UBV{n}) 

TP 

TP 

FP 

AV{n,})UBV{t}) 

TP 

TP 

FP 

AV{ni})  U  B\({n}) 

FN 

FN 

TN 

Table  3-14-  Across  II  OR  fused  classification  system  outcomes  with  respect  to  the  true  partitions  of  the  event  set. 


A111 

Ae 

B* 

tl2  =  {tl,  t2} 

ni=  {n,t3} 

t23  =  {t2,  t3} 

t 

n 

n23  =  {n,  ti} 

n 

n 

Table  3-15  -  Across  III  AND  combined  classification  system  truth  table. 


v'11 

Ae 

tl2  =  {tl,  t2} 

ni=  {n,t3 } 

t23  =  {t2,  t3} 

t 

t 

n23  =  {n,  tl} 

t 

n 

Table  3-16  -Across  III  OR  combined  classification  system  truth  table. 
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A111 

Ttl  (True 
target  type  I 
partition) 

rt2  (True 
target  type  II 
partition) 

rt3  (True  target 
type  III 
partition) 

Tn  (True  non¬ 
target  partition) 

aV  {ti2})  n  BVfe}) 

TP 

TP 

TP 

FP 

AV{ti2})  n  B\({n23}) 

FN 

FN 

FN 

TN 

AV{m2})  n  B\({t23}) 

FN 

FN 

FN 

TN 

AV{n12})  n  B\({n23}) 

FN 

FN 

FN 

TN 

Table  3-17  -  Across  III  AND  fused  classification  system  outcomes  with  respect  to  the  true  partitions  of  the  event  set. 


v/// 

Ftl  (True 
target  type  I 
partition) 

rt2  (True 
target  type  II 
partition) 

rt3  (True  target 
type  III 
partition) 

Tn  (True  non¬ 
target  partition) 

AV{tj2})  U  BVfe}) 

TP 

TP 

TP 

FP 

AV{tj2})  U  BV{n23}) 

TP 

TP 

TP 

FP 

AV{m2})  U  BVfe}) 

TP 

TP 

TP 

FP 

AV{ni2})  U  B\({n23}) 

FN 

FN 

FN 

TN 

Table  3-17  -  Across  III  OR  fused  classification  system  outcomes  with  respect  to  the  true  partitions  of  the  event  set. 

3.5  Probability  Theory  and  its  Applications  to  Classification  Theory 


In  order  to  create  a  ROC  curve  for  a  fused  classification  system,  one  must  choose 
whether  within  or  across  fusion  is  applicable  based  on  setting  and  then  apply  the  Boolean 
rule  of  choice.  Once  these  have  been  established,  there  exists  a  way  to  develop  a 
probability  expression  that  models  the  performance  of  the  classification  system,  namely, 
the  ROC  curve.  The  background  theory  presented  here  provides  a  way  to  derive  the  ROC 
curve  expression  for  within  and  across  label  fusion  using  only  the  performances  of  the 
individual  classification  systems.  It  is  assumed  that  classification  systems  Ae  and  B*  are 


54 


conditionally  independent  with  respect  to  the  non-target  event  set.  Independence  and 
conditional  independence  of  events  are  now  defined. 

Definition  3.11  Let  (F,  G,  Pr)  define  a  probability  measure  space.  Let  C  and  D  be  any 
sets  in  G.  Then  the  following  are  true: 

p{c  n  Dc)  =  p(c)  -  p(d  n  c) 

P{D  U  C)  =  P{D )  +  P(C)  -  P(D  n  C) 

Definition  3.12  (Independence  of  Events)  Let  (T,  G,  Pf)  be  a  probability  measure  space. 
The  collection  of  events  { Ej ,  E2,  ...,En}  c  G  is  said  to  be  independent  if 

\i=l  /  i=i 

Definition  3.13  (Conditional  Independence  of  Events)  Let  (T,  G,  Pr)  be  a  probability 
measure  space.  The  collection  of  events  {Ej,  E2,  ...,En}  a  G  are  said  to  be  conditionally 

independent  with  respect  to  event  Ej  if  for  any  sub-collect  ion^E^,  Ei2, ... ,  j  c 
{E1,E2,  ...,En]  then 

(m  \  m 

-nw 

i= 1  /  i=l 

3.6  Correlation 

In  {Schubert  et  al:  2005),  the  functions  for  the  ROC  curves  associated  with  the 
different  types  of  label  fusion  were  derived  under  the  assumption  that  classification 
systems  were  correlated.  Using  the  results  of  {Schubert  et  al:  2005)  correlation  between 
two  classification  systems  takes  the  functional  form  below. 
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(3.10) 


P\A-q>  Bn .] 


_ [Qab  ~  CACB] _ 

VQ(1-Q)VCb(1-Cb) 


Where 


cA  =  P(rt)pTP(Ae )  +  p(rn)pFP(Ae )  (3.ii) 

cB  =  p(rt)PTP(Bn)  +  p(rn)pFP(Bn) 

Qab  =  P(Tt)PTP(A9  A  Bn)  +  P(rn)PFP(Ae  A  Bn) 

Notice  that  Ca,  Cb,  Caab,  called  the  co.v?  functions,  are  weighted  functions 
involving  the  false  positive  and  true  positive  rates  that  constitute  the  ROC  curves  of 
classification  systems  Ae  and  B^.  They  arise  naturally  from  the  expressions  for  the 
expected  value  of  classification  systems  Ae  and  B*.  When  the  two  classification  systems 
are  independent,  the  correlation  term  goes  to  zero,  i.e.,  when  Caab  =  CaCb. 

Next,  the  functions  TA(p)  and  TB(q)  are  defined. 

Def  3.16  -  (T functions)  -  The  T functions  are  defined  to  be  the  cost  functions  for 
classification  systems  Aq  and  Bn  that  are  maximized  over  their  respective  parameter  sets. 
These  functions  are  the  optimal  performance  points  on  the  ROC  curves  of  classification 
systems  Ap  and  Bn,  respectively. 

TA(p)  =  ma ix  Q, 

=  max[P(Tt)P TP  (Ae)  +  P(rn)PFP(Ae)] 

=  ma x[P(rt)fA(p)  +  P(Tn)p ] 
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TB(q)  =  max  CBn 

=  max[P  (rt)P TP  (Bn)  +  P(rn)PFP(Bn )] 

n  G/7 

=  ma x[P(rt)fB(q)  +  P(rn)q] 

n  G/7 

These  functions  maximize  over  the  parameters  permitting  the  use  of  values  from 
the  ROC  curve  for  classification  systems  Ae  and  EC.  These  functions  take  the  same 
general  form  for  any  type  of  label  fusion,  however  the  prior  probability  weighting  on  the 
TP  and  FP  values  changes  depending  on  the  type  of  across  or  within  label  fusion  being 
applied. 

It  is  further  derived  in  {Schubert  et  al:  2005)  that  the  ROC  curve  for  any  AND 
label- fused  system  where  the  correlation  remains  constant  for  any  combination  of 
parameters  0  and  n  takes  the  following  form. 

,  ,  ,  1  ,  ,  ,.  ,  ^  PK)  (3.12) 

f c and (r)  =  ma xgp(TA(p).TB(q))  -  r 


The  value  “r”  is  the  product  of  FP  values  p  and  q  from  the  ROC  curves  of 
individual  classification  systems  Ae  and  EC  via  the  conditional  independence  assumptions 
on  the  non-target  partition  of  the  event  set.  The  function  gp  takes  the  following  form. 


9P  =  PJTaCpX1  -  TA(p)l  TB(q)(l  ~  TB(q )) 


(3.13) 


+  TA(p)TB(q ) 
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In  a  similar  fashion,  the  ROC  curve  for  any  OR  label-fused  system  under  the  assumption 
of  constant  correlation  between  the  classification  systems  at  each  parameter  combination 
may  be  defined. 


f  c0R (r) 


1 


— — —  max 

P\Tt)  p+q-pq=r 


hp{TA(p),TB(q )) 


P(rn) 

- r 

P(rt) 


(3.14) 


For  the  combined  OR  ROC  curve,  the  quantity  “r”  takes  the  form  of  the  OR  probability 
statement  for  the  false  positive  value  of  classification  system  AH  or  B*  and  hp  is  defined  as 
the  following. 


hp  —  TA(p)  +  TB(q) 


(3.15) 


-  PJrA(p)(i  -  ~ 

~  TA(p)TB(q ) 


For  a  more  in  depth  discussion  of  the  derivation  of  these  terms,  the  reader  is  directed  to 
(Schubert  et  al:  2005). 

3. 7  ROC  Curves  for  Different  Types  of  Label  Fusion 
3. 7.1  Within  Label-Fused  ROC  Curves 

Now  that  a  general  equation  for  a  label-fused  ROC  curve  has  been  developed,  it 
can  be  applied  to  the  different  cases  of  label  fusion.  First,  consider  the  within  AND 
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combined  classification  system.  As  elaborated  in  (Schubert:  2005)  the  combined  AND 
classification  system  under  constant  correlation  takes  the  following  form. 

Def  3.17  -  Within  AND  ROC  curve  -  Assume  that  the  ROC  curves  for  systems  A0  and  BK 
are  known.  Then  for  a  given  parameter  combination  6  and  n  the  (FP,  TP)  ordered  pair 
constitutes  a  point  on  the  ROC  curve  for  the  combined  classification  system  A  a  AND  Bn. 

Ptp(Aq  A  Bn)  =  — | —  max gp (Tff  (p),  Tjf  ( q ))  -  r 
P(Tt)pq=r  FV  P(Tt) 

pW(A0  A  Bn)  =  PFP(Ae)PFP(Bn) 


=  pq 


Where 


Pa'  (p)  =  ma ix[P(Tt)fA(p)  +  P(Tn)p ] 

UE0 

7 T(q)  =  ma x[P(Tt)fB(q)  +  P(Tn)PFP(Bn)] 

nen 

gp(T)f(p),Tif(q))  =  p  (Jtaw(P)(1  -  TAw(p))jTBw(q)(l  -  T™ (q))^  +  Tff  (p)T? (q) 


Clearly,  if  the  two  systems  are  uncorrelated,  then  the  correlation  constant,  p,  is 
zero  and  the  calculation  of  the  TP  value  becomes  the  product  of  the  T  functions  for 
classification  systems  Ae  and  B*  minus  the  weighted  FP.  Correlation,  by  definition,  is 
bounded  on  the  interval  [-1,  1]  where  p  =  -1  implies  the  two  classification  systems  are 
inversely  correlated  and  p  =  1  implies  the  two  classification  systems  are  directly 
correlated. 

The  within  OR  combined  classification  system  is  defined  in  a  similar  fashion. 
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Def  3.18  -  Within  OR  ROC  curve  -  Assume  that  the  ROC  curves  for  systems  An  and  Bn 
are  known.  Then  for  a  given  parameter  combination  6  and  n,  the  (FP,  TP)  ordered  pair 
constitutes  a  point  on  the  ROC  curve  for  the  combined  classification  system. 


P™{Ae  V  Bn) 


max 


P(Tt)  p+q-pq 


MT< 


w 

A 


ip)JB 


w 


(<?)) 


Pi.ru) 

- T 

P(rt) 


Pfp i^e  V  Bn)  —  PppiAg )  +  PppfBf)  —  PppiAp^PppfBf) 


=  p  +  q  -  pq 

Where 

Pa  ip)  =  max[P(rt)/4(p)  +  P(Tn)p] 

u  60 

T™(q)  =  max[P(rt)/B(g)  +  P(rn)q\ 

n  6/7 

hp{Tffip),T™{q)) 

=  Tf' ip)  +  T™iq)  -  p  (JtJTCpX  1  -  77(p))Jr^(c?)(  1  -  T^(q))^ 
-TAwip)T^iq) 

If  the  two  systems  are  uncorrelated,  the  hp  function  simplifies  to  the  form  of  the 
independent  OR  probability  calculation  using  the  T  functions. 

3. 7.2  Across  I  Label-Fused  ROC  Curves 

The  T  functions  for  the  across  I  label-fused  ROC  curves  closely  resemble  those  of 
the  within  label-fused  ROC  curves.  The  derivation  of  these  functions  can  be  found  in 
(< Schubert :  2005).  The  major  difference  between  across  and  within  label-fused  ROC 
curves  is  the  prior  probability  weighting  on  individual  classification  systems  Ag  and 
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Def  3.19  -  Across  I  AND  ROC  curve  -  Assume  that  the  ROC  curves  for  systems  Ag  and  Bn 
are  known.  Then  for  a  given  parameter  combination  6  and  n,  the  (FP,  TP)  ordered  pair 
constitutes  a  point  on  the  ROC  curve  for  the  combined  classification  system. 


Ptp^Ao  b  Bn)  — 


1 


—t—  max 
P(Tt)  pq=r 


gpiTiipfT^q)) 


P(Pn ) 

- T 

P(rt ) 


Pfp  (Ag  A  Bn)  —  Ppp  (Ag)PFP(Bn) 


=  pq 

Where 

rj(p)  =  max[p(rtJfA(p)  +  p(rnJp] 

t)E(y 

T‘(q)  =  max[p(rt2)/s(?)  +  P(r„2)q\ 

sNKp),  Th(q))  =  p  (JtJ(p)(1  -  Ti(p))jTi(q)(l  -  T^(q))J  +  Tj(p)T'(q) 

Note  that  the  prior  probability  associated  with  the  T  functions  has  changed  and  is 

dependent  upon  choice  of  label  fusion  rule. 

Next,  the  formula  for  the  across  I  OR  combined  ROC  curve  is  defined. 

Def  3.20  -  Across  I  OR  ROC  curve  -  Assume  that  the  ROC  curves  for  systems  Ag  and  B x 

are  known.  Then  for  a  given  parameter  combination  6  and  n,  the  (FP,  TP)  ordered  pair 

constitutes  a  point  on  the  ROC  curve  for  the  combined  classification  system. 

1  P(T ) 

P^pdAg  V  Bn)  =  ——  max  hp(Tidp)jUq))-—^-r 
P(lt)p+q-pq=r  H  PUt) 

Pfp (.Ag  V  Bn)  =  PFP (Ag)  +  PFP(Bn )  —  PFP(Ag)PFP(Bn ) 

=  p  +  q  -  pq 
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Where 

T’i(p)  =  max[P(rti)fA(p)  +  P(rn  )p] 

( 7C0 

Js(«)  =  max[P(rt2)/B(?)  +  P(r„2)?] 

htJTKpi.nun) 

=  Ti(p )  +  72(<7)  -  p  fjl) Cp)(l  -  Jj 1  - 

-Tl<P)Tl,(q) 

The  prior  probability  has  a  significant  effect  on  combined  ROC  curves  for  the 
across  I  combined  classification  system.  Performance  is  greatly  impacted  under  the  AND 
rule  as  there  are  few  instances  where  the  combined  classification  system  should  agree 
about  the  target  label  for  a  given  element. 

3.7.3  -  Across  II  Label-Fused  ROC  Curves 

In  the  section  below,  the  formulas  for  the  AND  and  the  OR  ROC  curves  are 
derived  for  the  across  II  combined  classification  system. 

Def  3.21  -  Across  II  AND  ROC  curve  -  Assume  that  the  ROC  curves  for  systems  Ag  and 

Bn  are  known.  Then  for  a  given  parameter  combination  9  and  n,  the  (FP,  TP)  ordered 

pair  constitutes  a  point  on  the  ROC  curve  for  the  combined  classification  system. 

1  P(T ) 

Ptp(Ao  b  Bn)  =  ——  max gp (Tfi (p),  Tfi (q))  - 

P\Tt)  pq=r  P\ft) 

Pfp(.Aq  A  Bn)  =  PFP (Ao)PFP(Bn) 


Where 


=  PI 


T"(P )  =  max[p(rti)/^(p)  +  P(rnJp] 
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Teiq)  =  max[P(Pt)/B(q)  +  P(Pn)q] 

n  G  77 

gP(Til(p),T‘l(q))  =  p  Qr"(p)(l  -  T''(p))jT'!(q)(l  -  T''(q)^  +  Tf  (p)T^(q) 

Note  that  the  prior  probabilities  associated  with  the  T  functions  of  system  B* 
under  across  II  label  fusion  are  identical  to  the  prior  probabilities  associated  with  system 
Bjj  under  within  label  fusion.  Hence,  across  II  fusion  becomes  within  fusion  when  the 
prior  probabilities  associated  with  system  A@  are  equivalent  to  the  prior  probabilities 
associated  with  system  Brc. 

Next,  the  across  II  OR  ROC  curve  formula  is  defined 
Def  3.22  -  Across  II  OR  ROC  curve  -  Assume  that  the  ROC  curves  for  systems  Ap  and  Bn 
are  known.  Then  for  a  given  parameter  combination  6  and  n,  the  (FP,  TP)  ordered  pair 
constitutes  a  point  on  the  ROC  curve  for  the  combined  classification  system. 

1  P(T ) 

P'MAg  VBn)  =— —  max  hp(Tl‘ (p),^1  (q))  - -j^-r 
PCPt)p+(?_p(?=r  P\.Tt) 

Pfp(.Aq  V  Bn)  =  PFP {Aq)  +  PFP(Bn)  —  PFP (A(j)PFP{Bn) 

=  p  +  q  —  pq 

Where 

Ta  (p)  =  max[p(TtJfA(p)  +  P(PnJp] 

at© 

T^fq)  =  max[P(rt)/B(q)  +  P(Pn)q] 

n  G  77 

hp  (Ta  (p)>  Tp  (q)) 

=  Tf  (p)  +  r"(q)  -  p  Qrj'(P)(i  -  rj'(p))  -  r"  (<?))) 


-ri'W(q) 


63 


3. 7.4  Across  III  Label-Fused  ROC  Curves 


Below,  the  ROC  curves  for  across  III  label-fused  systems  are  defined. 

Def  3.23  -  Across  III  AND  function  -  Assume  that  the  ROC  curves  for  systems  Ag  and  BK 

are  known.  Then  for  a  given  parameter  combination  6  and  n,  the  (FP,  TP)  ordered  pair 

constitutes  a  point  on  the  ROC  curve  for  the  combined  classification  system. 

1  P(T ) 

Ptp(Ao  A  Bn)  =  ——  max  gp  {T[“  (p),  Tfl  (q))  -  — p-r 
P{Tt)pq=r  H  P(Tt) 

Ptt(Ae  A  Bn)  =  PFP(Ae)PFP(Bn) 


=  pq 


Where 


Ti"(p )  =  max[P(rtl2)fA(p)  +  P(r„12)p] 
Tp!'(q)  =  ma x[P(r,2s)fB(q)  +  P{rnz3)q] 


=  P  (JT.!"<P><t  -  T!"(pj)J^"<qt(l  -  Tp"(q))j  +  Tj"(p)T'"(q) 

Def  3.24  -  Across  III  OR  ROC  curve  -  Assume  that  the  ROC  curves  for  systems  Ag  and  Bn 

are  known.  Then  for  a  given  parameter  combination  6  and  k,  the  (FP,  TP)  ordered  pair 

constitutes  a  point  on  the  ROC  curve  for  the  combined  classification  system. 

1  P(T  'l 

Ptp  G4<?  V  Bn )  =  — —  max  hp (Tf  (p),  Tjf1 (q))  -  — r 
P\,lt)v+q-vq=r  H  PUt) 


Pfp(Aq  V  Bn)  —  PFP (Aq)  +  PFP(Bn)  PFp(Ag)PFP(Bn) 


Where 


=  p  +  q  -  pq 
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Tl'\v)  =  max[P(rtl2)fA(p)  +  P(r„12)p] 

t)E(y 

n"w  =  ™ + p(rnjq] 


K(Tl"(p),Tl,"(q)) 

=  Ti"(v )  +  T'"(q)  -  p  H r"'(p)(l  -  T"'(p))Jt"'C</)(1  -  r"'(?))^ 
-Ti"(p)Ti"(q) 

In  section  IV,  the  results  of  the  research  will  be  discussed  along  with  derivations 
for  the  functions  that  describe  the  bias  between  across  and  within  label-fused 
classification  systems.  In  section  V,  the  simulation  will  be  analyzed.  This  simulation 
compares  the  ROC  curves  of  within  and  across  combined  classification  systems,  the  bias 
that  exists  between  these  ROC  curves,  and  different  environmental  factors  that  can  affect 
the  bias. 


65 


IV  Results 


Recall  that  the  aim  of  this  research  is  to  describe  and  quantify  the  bias  between 
classification  systems  under  the  assumptions  of  within  and  across  label  fusion.  The 
results  of  ( Schubert  et  al:  2005 )  suggest  that  there  is  some  inherent  difference  in  the 
performance  of  classification  systems  under  the  different  types  of  across  fusion  as 
compared  to  within  fusion.  First,  the  notion  of  bias  must  be  developed.  Generally,  bias 
will  be  defined  as  the  difference  in  true  positive  rate  for  a  given,  fixed  false  positive  rate. 
Thus,  bias  may  be  viewed  graphically  when  depicting  the  within  label-fused  ROC  curve 
and  the  across  label-fused  ROC  curve  on  the  same  plot  and  observing  the  vertical 
distance  between  the  two  curves  for  a  fixed,  x-axis  (false  positive)  value.  Label-fused 
ROC  curve  formulas  expressed  in  terms  of  the  performance  of  the  individual 
classification  systems  presented  previously  may  be  used  to  create  computational  formulas 
for  bias  in  terms  of  the  individual  systems.  Thus,  adjustments  to  fused  system 
performance  may  be  generated  in  a  flexible  manner.  Formulas  for  bias  between  within 
label-fused  ROC  curves  and  each  of  the  types  of  across  label-fused  ROC  curves  are 
derived  below. 

4.1  Bias  Between  Across  I  and  Within  ROC  Curves 

The  bias  between  across  I  and  within  label-fused  ROC  curves  may  be  expressed 
in  terms  of  the  fusion  rule  and  the  ROC  curves  of  the  individual  classification  systems. 
Definition  4.1.1  {Performance  Bias:  Across  I  versus  Within,  Boolean  AND  rule)  The 
performance  bias  between  fused  across  I  and  within  ROC  curves  under  the  Boolean  AND 
rule  is  defined  to  be  the  following. 
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BiCLSIDVSW  =  fWithin  AND  ^  _  jr  Across  I  AND 

BiasbVAS™  =  (m^[gp(TA  (p),Tg  (q))]  ~  ma=x[5p  (fj  (p '),  TB  (q '))]) 

It  is  possible  that  the  values  p  and  q  whose  product  produced  the  combined  false 
positive  rate,  r,  may  not  be  the  same  for  both  label-fused  ROC  curves.  Hence,  p  may  not 
be  equal  to  p’  and  q  may  not  be  equal  to  q’.  If  the  values  p  and  q  are  the  same  for  both 
label-fused  ROC  curves,  then  the  formula  can  simplify  slightly  to  the  following: 

Bias  'DVSJ  =  — -ma x[Tff (p)T’jf (q)  -  T^(p)T^(q)] 

P{rt)pq=r 

For  the  false  positive  values  p  and  q  to  be  the  same  for  across  and  within 
classification  systems,  the  combined  false  positive  rate  must  be  invariant  with  respect  to 
the  type  of  label  fusion.  That  is  to  say,  if  the  false  positive  values  are  calculated  in  a  way 
that  the  fusion  rule  has  no  bearing  on  the  computation  of  values  p  and  q  for  individual 
classification  systems,  then  this  assumption  can  be  met.  For  example,  in  the  simulation  of 
section  V,  the  values  for  p  and  q  are  generated  using  parameter  values  and  the  normal 
CDF.  Random  sampling  of  values  from  the  non-target  distribution  may  lead  to  instances 
where  p  9*  p’  or  q  i-  q’. 

The  following  two  theorems  are  concerned  with  deriving  the  formula  for 
performance  bias  between  the  across  I  and  within  label-fused  ROC  curves  under  the 
Boolean  AND  rule.  The  first  theorem  makes  no  assumption  about  the  equality  of  p,  p’,  q, 
and  q’.  The  second  theorem  is  a  simplification  when p  =  p’  and  q  =  q’. 

Theorem  4.1.1  (Performance  Bias:  Across  I  versus  Within  label-fused  ROC  curves, 
Boolean  AND  rule)  Let  D^AB  be  the  ROC  curve  for  the  Boolean  AND  label-fused  across  I 
system.  Let  D)fAB  be  the  ROC  curve  for  the  Boolean  AND  label-fused  within  system. 
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Assume  A  and  B  are  independent  classification  systems.  Then,  for  a  fixed  false  positive 
value,  max  pq  =  r  =  max  p  ’q  ’,  the  bias  between  across  I  and  within  label-fused  ROC 
curves  under  the  AND  rule  is 

BiasDZ^  =  (max[r/ (p)T^ (<?)]  -  max 

A!\a  r{lt)  \VQ=r  p'q'=r  ) 

Proof:  Assume  that  classification  systems  A  and  B  are  independent  and  the  correlation 
constant,  p,  is  zero.  Then,  bias  may  be  expressed  as: 

BiasIvsW  =  fwithin  AND  (r)  —  jrAcross  1 AND  (p^ 

=  tp).TB  (<?))  -  P(r„)r] 

-  -  p(fn)rl 

=  (max[r/ (p)T^(q)  -  P(rn)r]  -  max  -  P(rn)r]) 

rylt)  \pq=r  p'q'=r  / 

=  pfiT)  (max[(p(rt)prP<Ae)  +  P(rn)PFP{Ae)){P{rt)PTP(Bn) 

+  p(rn)pFP(Bn))] 

-  max  [(p(rtJpTP(Ae)  +  p(rni)pFP(Ae ))  (p(rt2)pTP(Bn) 

+  p(rn2)PFp(Bn))]) 

=  ^Tv  (maxtr/Cp)!^^)]  +  max  |Tj (p')ri (<?')]) 

ryit)  \pq=r  p'q'=r  / 

I 

Theorem  4.1.2  (Performance  Bias:  Across  I  versus  Within  label-fused  ROC  curves, 
Boolean  AND  rule  where  p=p  ’  and  q=q  ’)  Let  D^AB  be  the  ROC  curve  for  the  Boolean 
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AND  label-fused  across  I  system.  Let  D^AB  be  the  ROC  curve  for  the  Boolean  AND  label- 
fused  within  system.  Assume  A  and  B  are  independent  classification  systems.  Then,  for  a 
fixed  false  positive  value,  max  pq  =  r,  the  bias  between  across  I  and  within  label-fused 
ROC  curves  under  the  AND  rule  is 

Bias  =  jN  (ma x{TA"(p)T"(q)  -  Ti(p)Tj, (<,)]) 

Proof:  Assume  that  classification  systems  A  and  B  are  independent  and  the  correlation 
coefficient,  p,  is  zero.  Assume  the  probability  values  p  and  q  are  invariant  with  respect  to 
choice  of  label  fusion  rule.  Then,  bias  may  be  expressed  as: 

Bids  JDVS  W  =  f  Within  AND  ^y^  _  jr  Across  I  AND  ^y^ 

=  (max[5p(T7(p).C (<?))  -  Ptrn)r} 

~  mdx)igp{Tj(pr),TIB  (<?'))  - 

=  maX  [(p(rt)pTP(A0)  +  p(rn)pFP(Ae))(p(rt)pTP(Bn) 

+  p(rn)pFP(Bn)) 

-  p(rn)r-[[p(rtl)pTP(Ae)  +  p(rni)pFP(Ae))  [p(rtJpTP(Bn) 

+  p(rn2)pFP(Bn)))  +P(rny] 

=  p^max[PTP(Ae)PTP(Bn)(P(rtf  - P(rtJp(rt2)) 

+  Ptp{Aq)P  FP(Bn)  (P(Pt)P(Pn)  -  p(rtl)p(rnj) 

+  PFP{Ag)PTP(Bn)  (P(Pt)P(Pn)  -  p(rnJp(rt2)) 

+  PFP (A p)P Fp (Bf)  (P(P„)2  -  P(Pni)P(Pn2))] 
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P(rt)Pq= 


max [77 (p) 7^ (q)  -  T{(p)T!B(q)} 


In  a  similar  fashion  to  the  Boolean  AND  rule,  the  bias  formula  for  the  Boolean  OR 
rule  will  be  defined  and  then  the  corresponding  theorems  for  the  two  different  cases 
regarding  the  false  positive  values  will  be  derived. 

Definition  4.1.3  (Performance  Bias:  Across  I  versus  Within  Label-Fused  ROC  curves, 
Boolean  OR  rule)  Bias  between  the  Boolean  OR  label-fused  within  and  across  ROC 
curves  is 


Bias’ofW  =  f  Within  OR (r)  _  JT Across  /  OR  ^ 

=  P?7o(+maX  lhp(TA  (p)>tb  -  max  [hp(Tj(p'),T{,(q'))]) 

P(It)  \p+q-pq=r  pi+qi-piq,=r  r  / 

Theorem  4.1.3  (Performance  Bias:  Across  I  versus  Within  Label-Fused  ROC  curves, 
Boolean  OR  rule)  Let  DfB  be  the  ROC  curve  for  the  Boolean  OR  label-fused  across  I 
system.  Let  D%B  be  the  ROC  curve  for  the  Boolean  OR  label-fused  within  system. 
Assume  A  and  B  are  independent  classification  systems.  Then,  for  a  fixed  false  positive 
value,  max  p+q-pq  =  r  =max  p’+q’-p’q’,  the  bias  between  across  I  and  within  label- 
fused  classification  systems  is 
1  ( 


n;noI  vs  W  _ 
BlCLSDAvB  ~ 


P(rt ) 


(  max  [hp{Tff  (p),T^(q))\-  max  [^(^(p'),^^'))]) 

\p+q-pq =rL  r  J  pr+qr-prqr=rL  / 
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Proof:  Assume  that  classification  systems  A  and  B  are  independent  and  the  correlation 
coefficient,  p,  is  zero.  Then,  bias  may  be  expressed  as: 

1 


BiasoVII  = 


P(rt)  p+q—pq= 
1 


max  [hp(T^(p),T^ (<?))  -  P(Pn)r] 

n — n  n — r  ' 


P(rt)  p'+q'-p'q'=r ^  ^  ^  ^  n)  ] 

=  Tjrprr  (  max  [7/ ( p)T f  ( q )  -  P(Pn)r] 

PUt)  \p+q-pq=r 


~  ,  max  [Tj(p')7’B(<7')  -  P(rn)rh 

p'+q-p'q-r  ) 

=  ^U(  max  [7^(p)  +  T^Cq)  -  7^(p)7^(q)  -  P(r„)r] 

rylt)  \p+q-pq=r 

-  ,  max,  [7i(p')  +  7’i(q')-7’i(p')7’i((?')-JP(/;)r]>) 

p'+q'-p'q'=r  J 

=  Jft(  max  [(^(WpC^  +  ^CWpC^)) 

PUt)  \p+ci~pci=r 

+  (p(rt)pTP(Bn)  +  PCPJP^CBJ) 

-  (P(Pt)PrP(de)  +  P(Pn)PFP(^le))(P(Pt)PTP(B7T)  +  PCPjPppCB*))] 


—  max 

pr  +qr -pr qr  =r  L 


((p(r  tJPrpC^e)  +  p(rni)PFpUe)) 

+  (p(rt2)PTP(Bn)  +  p^JPppCsj) 

—  (^(^ti^TpC^e)  +  ^O^l^PpGh?))  (p(Pt2)^,7’p(^7r) 

+  p(rn2)PFP0?j))]) 

=  pi^>(  max  fapWfr).  (<?))]  -  max  [^(TiCp'),^ (<?'))]) 

r\it)  \p+q-pq=r  pr+qr-prqr=r  ^  / 


71 


Seeing  as  there  is  no  simplification  of  the  formula  beyond  attempting  to  group 
cross  terms,  it  was  decided  to  leave  the  result  in  the  form  of  the  h  function  for  ease  of 
interpretation. 

Theorem  4.1.4  (Performance  Bias:  Across  I  versus  Within  Label-Fused  ROC  curves, 
Boolean  OR  rule  where  p=p  ’  and  q=q  ’)  Let  D^VB  be  the  ROC  curve  for  the  Boolean  OR 
label-fused  across  I  system.  Let  D(fB  be  the  ROC  curve  for  the  Boolean  OR  label-fused 
within  system.  Assume  A  and  B  are  independent  classification  systems.  Then,  for  a  fixed 
false  positive  value,  max  p+q-pq  =  r,  the  bias  between  across  I  and  within  label-fused 
ROC  curves  is 

Biasi>A Is  =  ^p+max=r[/ip(r7(p),rf  (<?))  -  hp{Ti(.p),U(q))] 

Proof:  Assume  that  classification  systems  A  and  B  are  independent  and  the  correlation 
coefficient,  p,  is  zero.  Further,  assume  that  p  and  q  are  invariant  with  respect  to  choice  of 
label  fusion  rule.  Then,  the  bias  may  be  expressed  as: 

Bias  =  fWithin0R  (r)  -  fAcrossI  0R(r) 

=  F77F  max  [hp{Tff(p),T^(q'))-P(rn)r 

rUt)  P+Q-PQ=r 

-(K(T‘(p),T‘(q))-P(rn)r)] 

=  ~p7p~\  max  VJa  (P)  +  Tb  (?)  “  ta  (p)Tb  (<?)) 

r{lt)p+q-pq=r 

-  W(P)  +  TiCq)  -  T;(p)Ti(q))] 
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=  ——  max  f(P(rt)Prp(i4e)  +  P(rn)PFP(Ae)) 

*Ut)  P+Q-PQ=r  1 

+  (p(rt)Prp(Bff)  +  P(rn)pFP(Bn)) 

-  (p(rt-)pTP(Ae)  +  p(rn)pFP(Ae))(p(rt)pTP(Bn)  +  p(rn)pFp(sj) 

-  ((KrJp7TG40)  +  p(rni)pFp(iifl)) 

+  ( p(rt2)pTP(Bn )  +  p(r  „2)Pfp  (ptt)) 

_  (P(r ti)PTp(-^e>)  +  n,)PFpOC))  (P(Pt2)Prp(^Jr)  +  P(Pn2)PFp(^7r))^] 

=  max  [/ip  (7J  (p),  T™  (g))  -  hp  {T[  (p),  T qB  (<?))] 

PUt)  p+q-pct=r 

■ 

4.2  Bias  Between  Across  II  and  Within  ROC  Curves 

The  bias  between  across  II  and  within  label- fused  ROC  curves  may  be  expressed 
in  terms  of  the  fusion  rule  and  the  ROC  curves  of  the  individual  classification  systems. 
Definition  4.2.1  ( Performance  Bias:  Across  II  versus  Within  Label-Fused  ROC  curves, 
Boolean  AND  rule )  Bias  between  the  Boolean  d/VD  label-fused  within  and  across  II  ROC 
curves  is  defined  to  be  the  following 

BiaSp  vsW=  f  Within  AND  _  yr Across  II  AND  ^y^ 

=  (max[^p(r7(p).  tb  (<?))]  -  max  [5p(ri/(p'),^/(q'))]) 

Recall  from  section  III  that  classification  system  B^  under  the  across  II  fusion  rule 
is  designed  to  classify  all  target  types  in  the  environment.  As  such,  the  system  B„  is  the 
same  for  both  the  within  and  across  II  label-fused  systems.  That  is  to  say 

K(q)  =  Tgtf) 
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This  permits  a  nice  simplification  of  the  formula  when  q  =  q\ 

Theorem  4.2.1  (Performance  Bias:  Across  II  versus  Within  Label-Fused  ROC  curves, 
Boolean  AND  rule)  Let  DfAB  be  the  ROC  curve  for  the  Boolean  AND  label-fused  across 
II  system.  Let  Df/AB  be  the  ROC  curve  for  the  Boolean  AND  label-fused  within  system. 
Assume  A  and  B  are  independent  classification  systems.  Then,  for  a  fixed  false  positive 
value,  max  pq  =  r  =  max  p  ’q  ’,  the  bias  between  across  II  and  within  label-fused  ROC 
curves  is 

BiasD™BW  =  (max[^p(r7(p)<  T'b  (<?))]  -  maxJgp(Tifpf,Tf 

Proof:  Assume  that  classification  systems  A  and  B  are  independent  and  the  correlation 

coefficient,  p,  is  zero.  Then,  bias  may  be  expressed  as: 

Bias11  VsW  =  fWithi71  AND  _  yr  Across  II  AND 

=  ™x[sP{ta(pIK (<?))  -  p(rn)r] 

-  P(rn)r] 

=  -^-(max[Tf/(p)T^(q)  -  P(Tn)r]  -  max  [Tf  (p')T^q')  -  P(Tn)r]) 

t)  \pq=r  prqr=r  J 

=  pfifi  (max[(P(rt)PrP(Ae)  +  P(rn)PFP  (Ae))  (P(P :)PTP  (Bn) 

+  p(rn)pPP(Bn))-p(rn)r] 

-  max  [(p(rtl)PTP(Ae)  +  p(rnJpFP(Ae))  (P(rt)PTP(Bn) 

+  p(rjpFP(B*))-p(rn)r]) 
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=  (max[(P(rt)PrPG4e)  +  p(rn)pFP(Ae))(p(rt)pTP(Bn) 

+  p(rn)pPP(Bn))-i] 

-  max  [(p(rtl)pTP(Ae)  +  p(rni)pPP(Ae))  (p(rt)pTP(Bn) 

+  p(rn)pFP(s7T))]) 

=  (max[^p(r7(p)< tb  (<?))]  -  max[gp{T[l(p')J,B,{q'))^j 

■ 

Theorem  4.2.2  (Performance  Bias:  Across  II  versus  Within  Label-Fused  ROC  curves, 
Boolean  AND  rule  where  p=p  ’  and  q=q  ’)  Let  DfAB  be  the  ROC  curve  for  the  Boolean 
AND  label-fused  across  II  system.  Let  D^AB  be  the  ROC  curve  for  the  Boolean  AND 
label-fused  within  system.  Assume  A  and  B  are  independent  classification  systems.  Then, 
for  a  fixed  false  positive  value,  max  pq  =  r,  the  bias  between  across  II  and  within  label- 
fused  ROC  curves  is 

BiasoLSBW  =  fa)  W(P)  -  Ta(p))\ 

Proof:  Assume  that  classification  systems  A  and  B  are  independent  and  the  correlation 
coefficient,  p,  is  zero  and  the  values  p  and  q  are  invariant  with  respect  to  label  fusion  rule. 
Then,  bias  may  be  expressed  as: 

Bias1clvsw  =  AND  ^  _  JT  Across  II  AND 

=  ^ma x[gp(T]f  (p), T f  (<?))  -  P(P„)r] 

-  (PlTjfiq))  -  P(P„)r] 
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=  ^ma x[gp(T^ (p),  T f  (g))  -  gp{T[\p), 7^(g))] 

Substitute  in  TBW  where  appropriate 

=  -^-ma x[77(p)7^(g)  -  Tj'(p)r"(g)] 

P{Tt)pq=r 

=  — ^-max[r/ (p)r^ (g)  -  rj/(p)(r^ (g))] 

r{lt)pq=rL 

=  -  ri'(p))] 


The  bias  formulas  for  across  II  versus  within  combined  label-fused  systems  under 
the  Boolean  OR  rule  will  be  defined  analogously. 

Def  4.2.3  (. Performance  Bias:  Across  II  versus  Within  Label-Fused  ROC  curves,  Boolean 
OR  rule )  Bias  between  the  Boolean  OR  label-fused  within  and  across  II  ROC  curves  is 
defined  to  be  the  following 

BiaSp  vsW  =  f  'Within  OR  ^y^  _  Across  II  OR  ^y^ 


=  F7Fs{  max  [hp(TA(p')>K(q))]~  max  [hpiff 

rOt)  \P+q-pq=r  pi+qi-piqi=r  ) 

Theorem  4.2.3  (Performance  Bias:  Across  II  versus  Within  Label-Fused  ROC  curves, 
Boolean  OR  rule)  Let  DfVB  be  the  ROC  curve  for  the  Boolean  OR  label-fused  across  II 
system.  Let  Df)B  be  the  ROC  curve  for  the  Boolean  OR  label-fused  within  system. 
Assume  A  and  B  are  independent  classification  systems.  Then,  for  a  fixed  false  positive 
value,  the  bias  between  across  II  and  within  label-fused  ROC  curves  is 
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1  ( 


BiasoZW  (9))] 

max  [hpiji1  (p'XTj}  (V))]) 

pi+qi-p'qr=r  r  / 


Proof:  Under  the  assumption  of  independent  classification  systems  A  and  B,  the 


correlation  constant,  p,  is  zero.  Then,  bias  may  be  expressed  as: 

BiaSZSsW  =  f^ithin0R(r)  -  f  Across  U  OR 

1 


max  [/i,fc»,?W)-P(/;)r] 

P+Q-PQ=r 


-  P(rn)r] 


P  (rt)  p' +q' -P1  q' - 


l 

pUd 


(  max  [K(p)  +  T^(q)-T^(p)T^(q)-P(rn)r] 

\p+q-pq=r 


~  ,  max,  [7"(p')  +  TH{q')  -  T”  (p')T”  (q')  -  P(rn)r]) 

p'+q'-p'q'=r  ) 

=  Jft(  ,max  [(n/i)prp(4fl)  +  p(/;)pFp(^)) 

PUt)  yp+q—PQ-r 

+  (p(rt)pTP(Bn)  +  p(rn)PFp(Bn)) 

-  (p(rt)pTP(Ae)  +  P(rn)pFP (Ae))(p(rt)pTP (sj  +  p(rn)pFP(Bn))\ 


max 

p'+q'-p'q'=r 


)Ptp(Aq)  +  p{rni)PFP(Ae)^ 


+  (p(rt)PTP(Bn)  +  p(rn)PFP(Bj) 


-  (p(rtl)PTP(A0)  +  p(rni)PFP(Ag))  (p(rt)PTP(Bn)  +  p(rn)PFP(X)))]) 
=  p7t^(  max  [hp(TA(P)> tb  (<?))]  -  max  [hp(T^ (p'),T^ (<?'))]) 

r{lt)\p+q-pq=r  ^  pr+qr-prqr=r  ^  / 
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Theorem  4.2.4  (Performance  Bias:  Across  II  versus  Within  Label-Fused  ROC  curves, 
Boolean  OR  rule  where  p=p  ’  and  q=q  ’)  Let  DfVB  be  the  ROC  curve  for  the  Boolean  OR 
label-fused  across  II  system.  Let  be  the  ROC  curve  for  the  Boolean  OR  label-fused 

within  system.  Assume  A  and  B  are  independent  classification  systems.  Then,  for  a  fixed 
false  positive  value  max  p+q-pq  =  r,  the  bias  between  across  II  and  within  label-fused 
classification  systems  is 

BiasoZw  =  fi^p+™fq=r  K1  _  rBw(<?))(j7(p)  -  Tjfip))] 

Proof:  Assume  that  classification  systems  A  and  B  are  independent  and  the  correlation 
coefficient,  p,  is  zero.  Further,  assume  that  the  values  p  and  q  are  invariant  with  respect  to 
label  fusion  rule.  Then,  bias  may  be  expressed  as: 

BiaSBfSBW  =  f  Within  OR  ^  _  f  Across  II  OR 

=  fipj „+ may ( ta 0».  T«  («))  -  )r] 

=  Jpr  max  [77 ( p)  +  Tg (q)  -  T f  (p)T? (q) 

r  {It)  p+q~pq=r 

-  (Tfi  (p)  +  T'fq)  -  Tfi(p)Tfi(q))\ 

=  max  (p)  +  K (?)  -  T? (p)K  (?) 

PUt)  P+Q-Ptl=r  L 

-  (n!(.p)  +  (rjf  (q))  -  Tll(p)(T™(,q)))} 

=  ~p7p\  max  \Ta  (p)  “  ta  ( p)Tb  (<?)  -  ta  (p)  +  ta  ( P)T f  (q)] 

r(lt)p+q-pq=r 
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/,  V  iilUA 

r(/tJ  P+<J~P<J= 


max  [(Tjf  (p)  -  Tj7  (p))  -  (q)  (Tff (p)  -  Tfi  (p))] 

a-vci=r 


P(ft )  P+<Z-P<7= 


max_  [(i-rf((7))(r7(p)-r"(p))] 


4.3  Bias  Between  Across  III  and  Within  ROC  Curves 

The  bias  between  across  III  and  within  label-fused  ROC  curves  may  be  expressed 
in  terms  of  the  fusion  rule  and  the  ROC  curves  of  the  individual  classification  systems. 
Definition  4.3.1  ( Performance  Bias:  Across  III  versus  Within  Label-Fused  ROC  curves, 
Boolean  AND  rule )  Bias  between  the  Boolean  AND  label-fused  within  and  across  III 
ROC  curves  is  defined  to  be  the  following 

BiasjJ1  VSW  =  fWithin  AND  ^  _  yr Across  III  AND 

=  ^j(max[pp(r7(p).C (<?))]  -  n^Xr[gp(Till(p'),T‘ll(q'))]^ 

Theorem  4.3.1  (Performance  Bias:  Across  III  versus  Within  Label-Fused  ROC  curves, 
Boolean  AND  rule)  Let  Dlf)B  be  the  ROC  curve  for  the  Boolean  AND  label-fused  across 
III  system.  Let  DfhB  be  the  ROC  curve  for  the  Boolean  AND  label-fused  within  system. 
Assume  A  and  B  are  independent  classification  systems.  Then,  for  a  fixed  false  positive 
value,  max  pq  =  r  =  max  p  ’q  the  bias  between  across  III  and  within  label-fused  ROC 


curves  is 


r>inQIUVSW  _ 

BiclsdAab 


1  ( 


(ta  (pX  (9))]  -  JPq^r[3p(Ti"(p'),  Tb  7 ((?'))]) 


P(rt)  \m- 

Proof:  Assume  that  classification  systems  A  and  B  are  independent  and  the  correlation 
coefficient,  p,  is  zero.  Then,  bias  may  be  expressed  as: 
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BiasjJ1  VSW  =  f  Within  AND  ^  _  j: Across  III  AND 

=  ^jja x[gp(rF (p),Tb  (<?))  -  P(Tn>] 

"  pmaxr[5p(ri/7(p'), Ti"(qr'))  -  Wr>] 

=  ~pTp  \  (ma x[T^ (; p)T f  (q)  -  P(Pn)r]  -  max  [T[u (p')T^ 1  (qf)  -  P(rn)r]) 

rylt)  \pq=r  p'q'=r  J 

=  (max[(P(Pt)PrPC40)  +  p(rn)pFP(Ag))(p(rt)pTP(Bn) 

+  P(rn)PFP(Bn))]  -max  [(p(rtl2)PTP<iAe) 

+  P(rni2)PFp(Ae ))  (p(rt23)P„(B„)  +  P(^n23)PFp(57r))]) 

=  ^(max[(P(Pt)PTPG40)  +  P(rn)PFPUe))(P(Pt)PTP(57T)  +  P(rn)PFp(Bn))] 
-  max  [(p(Ptl)PTp(^)  +  P(rni)PFP(Ae ))  (p(rt23)PrP(PJ 
+  p(/;23)PfP(sjt))]) 


(max[#p(rf  ( p),Tb  (<?))] 

\pq=r 


max 

p'q'=r 


[aP(Ti"(p%Ti"w)]) 


Theorem  4.2.2  (Performance  Bias:  Across  III  versus  Within  Label-Fused  ROC  curves, 
Boolean  AND  rule  where  p=p  ’  and  q=q  ’)  Let  Df(B  be  the  ROC  curve  for  the  Boolean 
AND  label-fused  across  III  system.  Let  D]fAB  be  the  ROC  curve  for  the  Boolean  AND 
label-fused  within  system.  Assume  A  and  B  are  independent  classification  systems.  Then, 
for  a  fixed  false  positive  value,  max  pq  =  r,  the  bias  between  across  III  and  within  label- 
fused  ROC  curves  is 
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BiasDAAB  W  =  max[77 (p)r^(q)  -  Tjn(p)T£fl(q)] 

aab  P^lf.)  pq=r 

Proof:  Assume  that  classification  systems  A  and  B  are  independent  and  the  correlation 
coefficient,  p,  is  zero.  Further,  assume  that  the  values  p  and  q  are  invariant  with  respect  to 
choice  of  label  fusion  rule.  Then,  bias  may  be  expressed  as: 

Bids  111  vsW  =  j7 Within  AND  _  yr Across  III  AND 

=  (™*[9P{TAtv), tb  ( q ))  -  P^n)r] 

~  pmaxJ,gp  (T[u  (p'),  TlBn  (q'))  -  P(Pn)r]j 

=  maX  [{p(rt)pTP{Ae)  +  p(rn)pFP(A0))(p(rt)prP(5j 

+  P(Pn)PFP(B7T)) 

-  P(Pn)r-((p(rtl2)PrPC4p)  +  P{rni2)PFP(Ae))  (p{Tt23)PTP{Bn) 

+  P(Pn23)PFP(B7T}))  +P(Pn)r] 

=  max  [PTP(Ae)PTP(Bn)  (P(Pt)2  -  P(rtl2)p(rt23)) 

+  Ppp(Aq)P Fp(Bn)  (P(Pt)P(Pn)  -  P(rtl2)p(r„23)) 

+  PFP{Ae)PTP(Bn)  (P(Pt)P(Pn)  -  P(rni2)p(rt23)) 

+  PFP{Ae)PFP{Bn )  (P(P„)2  -  P(r„12)P(r„23))] 

=  -^-max[r7(p)rf  (q)  -  ^"(p)^)] 

P(rt)pq=r 
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The  bias  formulas  for  across  III  versus  within  combined  label-fused  systems 
under  the  Boolean  OR  rule  will  be  defined  analogously. 

Def  4.2.3  (. Performance  Bias:  Across  III  versus  Within  Label-Fused  ROC  curves, 

Boolean  OR  rule)  Bias  between  the  Boolean  OR  label- fused  within  and  across  III  ROC 
curves  is  defined  to  be  the  following 

BiaSp1  vsW  =  f  Within  OR  _  y  Across  II  OR 

=  p7f^(  max  \hp(TA  (v)Jb  (<?))  -  pirny] 

ri<lt)\p+q-pq=rL 

-  max  [hp{TiII{pr),TIBniqr))  -  P(Tn)r]) 

pi+qi-piqi=r  r  ) 

Theorem  4.2.3  (Performance  Bias:  Across  III  versus  Within  Label-Fused  ROC  curves, 
Boolean  OR  rule)  Let  DjfB  be  the  ROC  curve  for  the  Boolean  OR  label-fused  across  III 
system.  Let  DfyB  be  the  ROC  curve  for  the  Boolean  OR  label-fused  within  system. 
Assume  A  and  B  are  independent  classification  systems.  Then,  for  a  fixed  false  positive 
value,  max  p+q-pq  =  r  =  max  p’+q’-p’q’,  the  bias  between  across  III  and  within  label- 
fused  ROC  curves  is 

BiaSD^BW  =  f^){p+^-fq=J-hP  (Ta(P)>TB  (9))] 
max  [hp  (Tf,(p'),TfI 1  (<?'))]) 

pi+qi-piqi=r  J 

Proof:  Assume  that  classification  systems  A  and  B  are  independent  and  the  correlation 
coefficient,  p,  is  zero.  Then,  bias  may  be  expressed  as: 
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BiasF™w  = 


P(rt )  p+q-pq= 
1 


max  [hp(T^ (p), (q))  -  P(Pn)r] 

/nr — m  /nr  —  r  ' 


-  pTm  /  w,  [ftpW"(p,),7,i//(<7,))-^(/’nH 

rylt)p  +q  -p  q  -r 

=  -pTr~\  (  max  \Ta  (p)Tb  (<?)  “  P(Tn)r] 

rylt)  \p+q~pq=r 

~  ,  max,  [ri"(p')ri/7(q')-P(/;)r]) 

p'+q'-p'q'-r  ) 

=  WF^{  max  [T^Cp)  +  7^Cq)  - ^Cp)^^)  - P(r„)r] 

rylt)  \p+q-pq=r 

~  ,  max,  [rrCpO  +  ^CqO-^CpO^CqO-PCrjr]) 

p'+q-p'q-r  ) 

=  Jpr(  max  [(P(Pt)PTP(^)  +  P(rn)PFP(Ae)) 

PW)  \p+q-pq=r“ 

+  (P(Pt)Prp(P7r)  +  P(PJPfp(SJ) 

-  (p(rt)pTP(Ae)  +  p(rn)Ppp(^e))(p(pt)prp(p7T)  +  P(P„)PFp(src))] 
((p(rtJpTpUe)  +  p(rni2)PFp(Ae)) 

+  (p(rt23)PTP(Bn)  +  p(rn23)PFp(Bir)) 

-  (p{rt12)PTp(.Ae')  +  ^(rni2)Ppp(v4e)j  ^(/’t23)Prp(57r) 

+  P(^23)PFp(Blr)))]) 

=  p7F>  (  +max  TB  (<?))] 

r(7tJ  \p+q~pq=r 

max  [/ip('ri"(p,),7,B/(q'))]) 

pr+qr—prqr=r  / 


—  max 

p’ +qr  -p'  q’  =r 
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Theorem  4.2.4  (Performance  Bias:  Across  III  versus  Within  Label-Fused  ROC  curves, 
Boolean  OR  rule  where  p=p  ’  and  q=q  ’)  Let  D^B  be  the  ROC  curve  for  the  Boolean  OR 
label-fused  across  II  system.  Let  Df(B  be  the  ROC  curve  for  the  Boolean  OR  label-fused 
within  system.  Assume  A  and  B  are  independent  classification  systems.  Then,  for  a  fixed 
false  positive  value  max  p+q-pq  =  r  =  max  p  ’+q  ’-p  ’q  ’,  the  bias  between  across  III  and 
within  label-fused  ROC  curves  is 

BiasDAvBW  =  J^p+^xq=r\-hP^(p)’T^ 

Proof:  Assume  that  classification  systems  A  and  B  are  independent  and  the  correlation 
coefficient,  p,  is  zero.  Further,  assume  that  probabilities  p  and  q  are  invariant  with  respect 
to  choice  of  label  fusion  rule.  Then,  the  bias  may  be  expressed  as: 

Bids  111  vsW=  f  Within  OR  ^y^  _  yr Across  III  OR  ^y^ 

=  -  P^r 
~  {hp (Ta1  (p)<  Tr1  (q))  -  P(rn)r)] 

=  -±-  max  [(Tjf(.p)  +  TW(q)-T]f(.p)Tlf(q)) 

P(lt)  p+q-pq=r 

-  (Tffp)  +  T‘ll(q )  -  T{n(p)TlBn(q))\ 
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=  ——  max  Up(rt)PTP(Ae)  +  P(rn)PFP(Ae)) 

rUt)  P+Q~VQ=r  1 

+  C P(rt)pTP(Bn )  +  p(rn)pFP(Bn)) 

-  (p(rt)pTP(Ae)  +  p(.rn)pFP (Ae))(p(rt)pTP (sj  +  p(rn)pFP(Bn)) 

~  ((p{rtJpTp(Ae)  +  p(rnu)PFP(Ag )) 

+  (p(rt23)PTP(Bn)  +  p(rn23)PFP(Bn )) 

-  (p(rt12)PTp(Ae)  +  p(pn12)PFp(Ae))  (p(pt23 )PTp(Bn) 

+  P(.rn23)PFp(B Jr)))_ 

r(yt)  p+q-pq=r 

■ 

The  bias  formulas  that  are  concerned  with  the  across  I  and  across  III  label  fusion 
rules  are  very  similar  in  appearance.  It  will  be  seen  in  section  V  that  this  similarity 
extends  to  the  ROC  curves  themselves,  but  the  presence  of  the  common  type  2  target  in 
across  III  fusion  will  lead  to  different  results  as  compared  to  across  I  label  fusion. 

In  the  following  section  the  simulation  that  was  produced  will  be  discussed  and 
the  results  analyzed.  Finally,  in  section  VI,  the  discussion  will  highlight  those  results  that 
were  discovered  through  this  work. 
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V  Simulation 


A  computer  simulation  was  used  to  determine  the  extent  of  performance  bias  that 
may  exist  between  classification  systems  that  use  within  label  fusion  as  opposed  to  across 
label  fusion.  In  addition,  individual  classification  system  performance,  correlation,  and 
target  prevalence  were  varied  in  order  to  establish  how  these  factors  affect  any  potential 
bias.  All  coding  for  the  simulation  was  developed  using  MATLAB®. 

5. 1  Construction  of  the  Simulation 

5.1.1  Simulated  Classification  Systems 

Assume  that  classification  systems  Ae  and  exist  and  are  the  same  mapping 
compositions  as  outlined  in  section  3.1.2.  In  this  instance,  the  parameter  set,  n,  is  a  two 
dimensional  set  composed  of  parameters  y  and  s.  Define  the  parameter  sets  0  =  [-4,  6] 
and  the  parameters  sets  y  =  [-4,  6]  and  s  =  [0,  10].  Assume  that  the  features  of  the  non¬ 
target  distribution,  ®n,  is  distributed  as  N(0,  1).  Also,  assume  that  the  features  of  the 
target  distribution,  ®t,  is  normally  distributed  with  variance  of  1 .  Let  the  two 
classification  systems  be  defined  thusly: 


ft  : 

X  E  rt;  X  > 

ae  -  In  : 

otherwise  J 

(5.1) 

,  -  (t: 

x  6  rt;  y  <  x  <  (y  +  £) j 

r’£  In  : 

otherwise  J 

Hence  for  classifier  a@,  if  the  element  from  the  feature  set  is  greater  than  or  equal  to  the 
parameter  value  0,  then  the  classification  system  gives  the  element  a  “f  ’  label.  If  not,  the 
element  is  given  an  “n”  label.  For  classifier  b7-£,  if  the  element  from  the  feature  set  is 
between  the  parameter  value  y  and  the  sum  (y  +  s),  then  the  element  receives  a  “f  ’  label 
and  an  “n”  label  otherwise. 
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5.1.2  Area  under  a  ROC  Curve 


In  order  to  examine  the  effects  of  individual  classification  system  performance  on 
any  bias  that  exists  between  within  and  across  fused  systems,  area  under  the  ROC  curve 
(AUC)  was  used.  AUC  has  a  specific  interpretation  in  that  a  perfect  classification  system 
has  an  AUC  of  1  whereas  an  equivalently  random  classification  system  has  an  AUC  = 
0.5.  Any  ROC  curve  with  AUC  less  that  0.5  performs  worse  than  chance.  Thus,  by 
exploiting  the  statistical  properties  of  the  AUC,  specific  classification  system 
performances  may  be  determined.  Using  these  properties,  it  will  be  assumed  that  the 
target  and  non-target  partitions  from  the  feature  set  come  from  different  N(p,  a  ) 
distributions.  Assume  that  there  are  three  levels  of  classification  system  performance: 
good,  fair,  and  poor.  A  good  classification  system  will  have  a  corresponding  AUC  of 
0.95,  a  fair  classification  system  will  have  an  AUC  of  0.85,  and  a  poor  classification 
system  will  have  an  AUC  of  0.75.  Fixing  system  performance  by  these  three  levels  of 
AUC  implies  specific  distributions  for  the  features.  Hence,  there  exists  a  way  to  solve  for 
the  mean  and  standard  deviations  of  the  distributions  for  targets  and  non-targets  that 
generates  the  ROC  curve  with  specific  AUC. 

Definition  5.1  (Normal  Area  under  a  ROC  curve)  Let  a  =  (p+  -  p.)  (the  difference 
between  the  mean  of  the  target  distribution  and  non-target  distribution).  Let  b  =  (a./  o+) 
(the  ratio  of  the  standard  deviation  of  the  non-target  distribution  over  the  standard 
deviation  of  the  target  distribution).  Let  &'1  be  the  inverse  Normal  CDF.  Then  the  area 
under  an  ROC  curve  for  normally  distributed  event  partitions  is 
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AUC  =  <2>_1 


=  0-i 


0*+  ~  /*-) 

\FWJ 


Let  the  distribution  of  the  non-targets  be  N(0,  1).  Thus,  for  a  good  classification 
system,  pt  =  2.326.  For  a  fair  classification  system,  pt  =  1.465,  and  for  a  poor 
classification  system  |Lit  =  0.954.  The  classification  systems  defined  in  5.1.1.  were 
constructed  specifically  so  that  each  of  the  three  previously  described  levels  of 
performance  using  AUC  criteria  could  be  established  with  the  target  means  as  defined 
above. 


5.1.3  Simulation  Scenarios 

The  probability  of  true  positive  (TP)  and  false  positive  (FP)  for  classification 
systems  A@  and  B7  e  is  defined  to  be  the  following. 


PTPiAe)  =  P{Al{At)\rt)  = 


p(Ai(At)nrt ) 


P(.rt) 


-J 


1 


PpptAe )  =  P{A^At)\rn)  = 


P{Ao(At)  n  rn) 

P(rn ) 


f2n 

v 

oo 

=  f- 

J  fll l 


PTp  iPy.e)  =  PW'MtWt)  = 


n  rt) 


e 

(y+£) 


t)ortj_  r  l 

})  J  V2 


Pfp(by,£ )  =  p{BlMt)\rn)  = 


Pi.Pt) 

p(B£eiAt)  n  rn) 
PiPn) 


(y+£) 


j 


2  dx 

■M2 

-e 

i 

2  dx 

-(x-fit)2 

—  e 
n 

2 

1 

-w2 

yf2n 


e  2  dx 


Y 

Using  Matlab  programming  software,  these  values  and  their  corresponding  ROC 
curves  were  generated  for  each  of  the  two  Boolean  rules  (Boolean  AND  and  OR)  and  for 
each  of  the  three  combinations  of  performances  for  A@  and  (toggling  between  good, 
fair,  and  poor)  as  determined  by  AUC.  In  addition  to  varying  the  performance  of 
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competing  classification  systems,  the  effects  of  correlation  between  classification  systems 
were  examined.  Seven  different  levels  of  correlation  were  considered;  -0.8,  -0.5,  -0.3,  0 
(independence),  0.3,  0.5,  and  0.8.  It  was  also  of  interest  to  investigate  the  effects 
associated  with  altering  target  populations.  Three  target  populations  were  investigated,  a 
target  rich  population,  a  target  enhanced  population,  and  a  target  deficient  population.  In 
the  target  rich  environment,  probability  of  observing  a  target  event  is  P(Tt)  =  4/5  and  the 
probability  of  observing  a  non- target  event  is  P(Tn)  =  1/5.  In  the  target  enhanced 
environment,  the  probability  of  observing  a  target  event  P(Tt)  =  2/3  and  the  probability  of 
observing  a  non-target  event  is  P(Tn)  =  1/3.  Finally,  in  the  target  deficient  environment, 
the  probability  of  observing  a  non-target  event  is  P(Tt)  =  1/5  and  the  probability  of 
observing  a  non-target  event  is  P(Tn)  =  4/5. 

5.1.4  Algorithm  Outline 

In  this  section,  a  brief  outline  of  the  algorithm  is  given.  Both  the  source  code 
comments  are  given  in  appendix  C. 

1)  Create  parameters  0,  y,  and  s 


2)  Generate  the  probabilities  of  true  and  false  positive  at  each  parameter  value  for 
individual  systems  Ae  and  Byx. 


for  i  =  1:  N 

OO 

Ptp(A9(i))=  I 


1  zfzEil 
e  2  dx 


0(0 

OO 


V27T 


pfp(a0(  o)=  f-4 

0( 


~(x)2 

e  2  dx 


end 
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for  i  =  1:  N 
for  j  =  1:  N 


y  (0+£(/) 


PTP  (fl^Gj))  —  J 


1  -(*~/u)2 

e  2  dx 


y(0 
rCO +£(;') 


\[2n 


PFp  —  f 

r(0 


1  ~(x)2 

e  2  dx 


y[2n 


end 

end 


3)  The  ROC  curve  for  systems  Ae  and  By  ;:  are  computed  from  the  probabilities  of  2). 


fAg  =  max[P  FP  (A  o)  PTp(Ae)] 

fBre  =  ma  x[Pfp(BY£)  PTp(BY:£)] 

y’ 

s€E 

4)  Input  the  true  positive  and  false  positive  values  associated  with  the  ROC  curves 
for  systems  AH  and  B7i,:  into  one  of  the  ROC  curve  formulas  for  either  within  or 
across  fusion  to  generate  the  label-fused  ROC  curve  for  the  within  and  across 
combined  systems. 


A  =  sotd.  B  =  d«i»i 


09  1 


The  figures  above  are  examples  of  ROC  curves.  For  the  figure  on  the  right,  the 
disparity  in  true  positive  rate  (height  of  the  curve)  is  the  performance  bias 
between  label-fused  ROC  curves.  ROC  curves  were  generated  for  each  possible 
combination  of  performance,  prevalence,  and  target  environment.  They  are 
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supplementary  material,  but  as  the  focus  of  this  document  is  in  the  performance 
bias  between  ROC  curves,  the  ROC  curves  themselves  will  not  be  discussed 
further  here.  The  ROC  curves  are  catalogued  in  appendix  A. 

5)  Compute  the  performance  bias  for  a  given  Boolean  rule,  a  given  level  of 
performance  for  classification  systems  Ae  and  Byx,  and  positive  or  negative 
correlation.  Each  output  consists  of  four  curves  depending  on  whether 
investigating  positive  or  negative  correlation  coefficients. 


6)  Plot  bias  versus  false  positive  rate  to  create  the  bias  curve  between  within  and 
across  label-fused  ROC  curves.  The  bias  curve  graphically  depicts  the  difference 
in  true  positive  rate  at  every  combined  false  positive.  The  figures  above  are 
examples  of  these  bias  curves. 

5.2  Correlation  and  its  Effects  on  Bias  between  Across  and  Within  Combined 
Classification  Systems 

In  the  prior  section,  all  of  the  formulas  for  the  bias  were  derived  under  the 
assumption  that  the  two  systems  were  uncorrelated.  The  cost  functions  defined  in  section 
III  were  derived  under  the  assumption  of  a  fixed  level  of  correlation.  In  the  case  that  the 
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two  classification  systems  are  indeed  independent,  the  correlation  coefficient,  p,  is  zero. 
Other  values  of  the  correlation  coefficient  were  investigated  to  see  the  effect  on 
performance  bias  between  label-fused  ROC  curves. 

Recall  from  section  III  that  the  correlation  between  classification  systems  Ae  and 
B,  ,:  was  defined  as: 


P\A-Q>  ^Tt\ 


_ [Caab  ~  Qfg] _ 

jcAa-cAycBa-cB) 


(5.1) 


The  correlation  expression  is  built  into  the  gp  and  hp  functions  (3.13  and  3.15, 
respectively).  Six  different  levels  of  non-zero  correlation  coefficient  values  were  chosen 
such  that  a  trend  may  be  seen  in  the  corresponding  bias  curve  output. 


Table  5-18  -  Different  levels  of  correlation  to  be  tested. 

In  the  context  of  this  document,  a  ROC  curve,  regardless  of  the  level  of 
correlation,  can  never  attain  a  true  positive  rate  greater  than  the  value  “1.” 
Simultaneously,  a  lower  bound  must  be  placed  on  the  true  positive  rate  when  negative 
correlation  is  considered.  As  it  is  not  possible  for  a  probability  to  be  less  than  zero,  the 
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lower  bound  on  the  true  positive  rate  is  the  max  of  either  zero  or  the  value  returned  by  the 
ROC  function.  These  bounds  may  be  defined  as: 


max(/c(r))  =  min{l,/c(r)} 

r  r 

min(/c(r))  =  max{0,/c(r)} 

r  r 


(5.2) 


In  the  following  subsections,  the  effects  of  altering  probability  weights  for  across 
specific  targets,  altering  the  level  of  dependence  between  individual  classification 
systems,  and  altering  target  environment  are  investigated  and  their  effect  on  performance 
bias  is  analyzed. 

5.2.1  -  The  Effect  of  Correlation  on  Bias  between  Across  I  and  Within  Combined 
Classification  Systems 

The  difference  in  performance  between  across  I  label-fused  systems  as  compared 
to  within  label-fused  systems  can  be  significant.  Recall  that  in  the  context  of  across  I 
label  fusion,  it  is  assumed  that  there  are  two  mutually  exclusive  target  types  that 
classification  systems  Ae  and  are  tuned  to  classify.  Under  this  assumption,  the 
likelihood  that  both  classification  systems  Ae  and  B7;£  label  the  same  element  in  question 
with  a  target  label  is  quite  low;  particularly  given  that  those  target  types  to  which  Ae  and 
B  ,  ;:  are  not  tuned  (ti  and  t2,  respectively)  have  been  scaled  to  the  non-target  N(0,  1) 
distribution.  On  the  other  hand,  under  the  assumption  of  within  label  fusion,  the  target  set 
for  both  individual  classification  systems  Ae  and  BY>£  is  the  same  and  constitutes  all 
element  with  target  features  in  the  event  set. 
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In  this  simulation,  the  probability  weights  associated  with  the  within  label-fused 
classification  system  are  equivalent  to  the  probability  of  observing  any  target  event  (as 
determined  by  target  environment).  For  the  across  I  label-fused  classification  system,  it 
was  decided  that  two  cases  should  be  considered:  1)  the  probability  of  observing  a  ti 
event  is  equal  to  the  probability  of  observing  a  t2  event  2)  the  probability  of  observing  a  ti 
event  is  not  equal  to  the  probability  of  observing  a  t2  event.  This  alteration  of  individual 
target  probability  gives  the  experimenter  the  ability  to  change  the  performance  of  the 
across  I  system  without  altering  the  within  label-fused  system.  In  section  5.2,  all  of  the 
cases  investigated  here  were  simulated  in  the  target  enhanced  environment.  The  other 
two  environments  will  be  analyzed  in  section  5.3.  Not  all  figures  will  be  presented  here. 
For  a  complete  listing  of  the  figures,  refer  to  appendix  B. 

5.2.1.1  -  Effect  on  Bias  when  p>0  under  the  AND  rule 
Suppose  that  two  classification  systems  are  highly  positively  correlated.  Given 
that  this  is  the  case,  it  would  be  reasonable  to  believe  that  the  two  classification  systems 
under  the  AND  rule  may  exhibit  increased  performance.  There  is  a  clear  trend  that  arises 
from  the  investigation  of  the  bias  between  across  I  versus  within  correlated  classification 
systems  under  the  AND  rule:  as  the  correlation  between  classification  systems  Ae  and  B7;£ 
increases,  the  performance  bias  decreases. 

It  was  hypothesized  that  altering  both  target  prevalence  and  classification  system 
performance  should  have  an  effect  on  the  level  of  bias  between  within  and  across  label- 
fused  ROC  curves.  Consider  first  the  case  that  both  classification  systems  have  equal 
levels  of  performance  (both  systems  are  either  good,  fair,  or  poor)  and  the  probability 
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weighting  associated  with  across  target  types  is  equally  distributed  among  the  two 
classification  systems  under  the  across  I  label  fusion  rule. 


0.35 


Bias  between  A1  vs  W  A=B=fair  pos  correlations 


0.25 


Bias  between  A1  vs  W  A=B=poor  pos  correlations 


max  pq=r  (c4  =  solid.  c5  =  -  -.  c6  =  :.  c7  =  x) 


Bias  between  A1  vs  W  A=fair  B=good  pos  correlations 


0  4 


Figure  5-1  -  Bias  curves  across  I  and  versus  within  AND.  The  y  -  axis  is  a  measure  of  the  bias  between  the  two 
classification  systems  at  a  fixed  fpr  (x  -  axis).  In  the  three  graphs  above,  classification  systems  A0  and  BY  £  are  fused 
at  equal  levels  of  performance.  For  reference,  P(rtl)  =  P(rt2)=  1/3  and  P(rnl)  =  P(rn2)  =  2/3  under  the  across  I  label 

fusion  rule. 


Altering  the  performance  of  individual  classifications  indeed  plays  a  role  in  the 
magnitude  of  performance  bias  between  different  label-fused  ROC  curves.  As  is 
demonstrated  in  figure  5-1,  as  the  level  of  performance  of  classification  systems  Ae  and 
BT;£  decreases,  the  difference  in  performance  between  the  types  of  label  fusion  also 
decreases.  The  change  in  performance  for  the  within  label-fused  ROC  curves  is  more 
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pronounced  whereas  the  across  I  label-fused  curves  aren’t  performing  much  better  than 
chance  even  at  the  good  level.  This  makes  sense  for  the  across  I  system  as  Ae  and  BY;£  are 
tuned  to  mutually  exclusive  target  types,  so  it  is  unlikely  that  the  systems  will 
simultaneously  label  any  target  event  with  the  target  label. 

Correlation  also  has  a  clear  effect  on  the  bias  between  fused  ROC  curves.  As  the 
level  of  positive  correlation  increases  (the  dashed,  dotted,  and  bold  curves  in  the  figures), 
notice  that  under  the  AND  rule  the  level  of  bias  between  across  I  and  within  label-fused 
systems  decreases  appreciably.  It  is  arguable  that  this  increase  in  performance  of  both 
systems  is  the  result  of  redundant  information  given  that  the  individual  classification 
systems  AH  and  BY;£  are  positively  correlated. 

Next,  consider  the  case  where  two  classification  systems  are  being  fused  under 
the  AND  rule  where  the  level  of  performance  is  different  for  each  individual  system. 


Bias  between  A1  vs  W  A=fair  B=good  pos  correlations 


Figure  5-2  -  Fusing  two  classification  systems  that  do  not  have  equal  performance.  The  level  of  bias  between  the 
fused  systems  is  most  heavily  influenced  by  the  superior  individual  classification  system. 


Surprisingly,  if  difference  in  performance  of  two  classification  systems  differs  by 
±  0.20  AUC,  the  level  of  bias  between  across  I  and  within  combined  classification 
systems  remains  largely  unchanged  with  respect  to  the  superior  classification  system.  For 
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example,  consider  figure  5-2  where  the  performance  of  classification  system  AH  is  fair 
and  B,,x  is  good.  The  amount  of  bias  between  the  across  I  and  within  AND  label- fused 
ROC  curves  is  comparable  to  that  when  both  systems  have  good  performance. 

Next,  suppose  that  the  prior  target  prevalence  for  classification  systems  A@  and 
By  g  is  altered  such  that  the  probability  of  observing  a  type  I  target  is  greater  than  the 
probability  of  observing  a  type  II  target. 


Figure  5-3  -  Altering  the  target  prevalence  for  classification  systems  A0  and  BY/E  under  across  I  label  fusion  has  a 
considerable  effect  on  the  bias  between  systems.  In  this  case,  P(rtl)  =  1/2  and  P(rt2)  =  1/6. 

Perhaps  the  most  unusual  aspect  of  the  change  in  target  prevalence  is  the  effect  it 
has  on  the  bias  curves.  In  the  case  where  target  prevalence  was  distributed  equally 
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between  the  two  classification  systems,  the  shape  was  generally  smooth  and  concave.  The 
magnitude  of  bias  between  the  within  and  across  I  combined  classification  systems  under 
the  AND  rule  decreases  when  performance  levels  of  individual  classification  systems  are 
equivalent.  By  shifting  the  majority  of  probability  weight  onto  either  ti  or  1 2  for  the 
across  I  fused  system,  it  is  acting  more  like  a  within  classification  system.  This  is  only 
true  when  the  classification  system  tuned  to  the  target  type  with  the  majority  of  target 
weight  performs  at  a  level  greater  than  or  equal  to  the  opposing  system.  The  opposite  is 
true  when  the  system  with  the  majority  of  target  weight  performs  at  a  lower  level.  In  the 
following  figure,  the  probability  of  observing  a  ti  event  is  greater  than  the  probability  of 
observing  a  t2  event. 


Bias  between  A1  vs  W  A=good  B=poor  pos  correlation  une<*ial  priors  Bias  between  A1  vs  W  A=poor  B=good  pos  correlation  unequal  priors 


Figure  5-4  -  If  system  A0  has  superior  performance  to  system  Bv  e/  the  bias  between  across  I  and  within  and  fused 
systems  can  be  greatly  decreased.  If  system  A0  performs  poorly,  regardless  of  the  performance  of  system  Bv#e,  the 
fused  across  I  and  classification  system  will  be  severely  influenced  by  system  A0.  In  this  case,  P(rtl)  =  1/2  and  P(rt2)  = 

1/6. 

If  the  performance  of  A@  is  good  when  the  performance  of  Byx  is  poor  (fig  5-4, 
left  hand  side),  the  magnitude  of  bias  between  within  and  across  I  ROC  curves  reaches  a 
maximum  of  roughly  ±0.20  tpr  (p  =  0).  If  the  performance  of  Ae  is  poor  when  the 
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performance  of  B7;E  is  good  (figure  5-4,  right  hand  side)  the  maximum  level  of  bias  can 
be  as  high  as  approximately  ±0.55  tpr  (p  =  0).  Even  in  the  case  that  both  classification 
systems  are  significantly  positively  correlated  (p  =  0.8),  the  bias  between  the  two 
classification  systems  can  still  be  as  high  as  ±0.40.  This  is  to  say  that  altering  target 
weights  for  across  I  targets  can  be  a  useful  tool  for  manipulating  the  performance  bias. 

5. 2.1. 2  -  Effect  on  Bias  when  p  <0  under  the  AND  rule 
When  two  classification  systems  are  negatively  correlated,  they  are  said  to  be 
inversely  related.  First  consider  two  classification  systems  with  equal  levels  of 
performance  and  equally  distributed  target  weight  (for  the  across  system).  If  both 
classification  systems  have  good  performance  (fig  5-5,  bottom),  it  is  seen  that  there  is 
only  a  minor  relationship  between  negative  correlation  of  classification  systems  and 
performance  bias  between  within  and  across  I  label-fused  ROC  curves.  Negative 
correlation  has  small  and  erratic  effects  on  the  performance  bias  between  label-fused 
ROC  curves  under  the  Boolean  AND  rule.  The  results  from  the  simulation  suggest  that 
the  only  noticeable  difference  in  bias  between  within  and  across  I  AND  label-fused  ROC 
curves  occurs  when  p  =  0  and  appears  more  or  less  constant  for  all  tested  levels  of 
negative  correlation. 
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Figure  5-5  -  Negative  correlation  has  a  minor  influence  on  the  level  of  bias  between  within  and  across  I  fused 
classification  systems.  For  reference,  P(rtl)  =  P(rt2)=  1/3  and  P(rnl)  =  P(rn2)  =  2/3  under  the  across  I  label  fusion  rule. 

If  the  performances  of  individual  systems  are  not  the  same,  then  it  is  reasonable  to 
suspect  that  there  may  be  a  noticeable  shift  in  the  bias  between  classification  systems. 
However,  the  simulation  provided  no  evidence  to  support  this  hypothesis.  Results  from 
the  simulation  strangely  suggests  that  when  the  performances  of  classification  systems  Ae 
and  B,,x  are  different,  any  value  from  p  =  0  to  p  =  -0.8  has  no  effect  on  the  performance 
bias  between  fused  within  and  across  I  label-fused  ROC  curves  (fig  5-6).  It  was  also 
tested  to  see  whether  altering  the  weighting  of  across  I  target  types  affected  bias  when 
the  individual  systems  were  negatively  correlated  under  the  Boolean  AND  rule. 
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Bias  between  A1  vs  W  A=far  B=good  neg  correlations 


Bias  between  A1  vs  W  A=faw  B=poor  neg  correlations 


0.3 1 


Figure  5-6  -  When  two  classification  systems  are  fused  with  differing  performance,  the  presence  of  negative 

correlation  has  little  to  no  effect. 

Altering  the  weights  of  across  I  target  types  demonstrates  that  negative 
correlation  does  have  some  effect,  but  it  is  minor  under  the  Boolean  AND  rule.  In  figure 
5-7,  it  is  seen  that  that  are  small  differences  in  the  performance  bias  when  systems  A@  and 
By-g  are  performing  at  different  levels,  but  hardly  pronounced  enough  to  see  visibly. 
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Bias  between  A1  vs  W  A=B=poor  neg  correlations  unequal  pnors 


Bias  between  A1  vs  W  A=good  B=poor  neg  correlations  unequal  priors  Bias  between  A1  vs  W  A=poor  B=good  neg  correlations  unequal  priors 


Figure  5-7  -  By  increasing  the  prevalence  of  tlf  the  influence  of  negative  correlation  becomes  easier  to  identify. 

Though  the  effects  are  minimal,  under  the  and  rule,  the  level  of  bias  does  decrease  between  within  and  across  I 

systems.  In  this  case,  P(rtl)  =  1/2  and  P(rt2)  =  1/6. 

5.2.13  -  Effect  on  Bias  when  p>0  under  the  OR  rule 
The  results  provided  by  the  simulation  suggest  that  positive  correlation  has 
minute  effects  on  the  bias  between  label-fused  OR  classification  systems.  This  is  a 
strange  parallel  with  the  negatively  correlated  label-fused  AND  curves.  What  can  be  said 
about  the  performance  bias  between  label- fused  OR  curves  is  that  the  magnitude 
decreases  at  an  accelerated  rate  as  compared  to  the  AND  label-fused  ROC  curves.  This  is 
largely  due  to  the  fact  that  within  label-fused  curves  under  the  OR  rule  reach  a  maximum 
level  of  performance  (tpr  =  1)  at  a  lower  corresponding  false  positive  rate. 
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Bias  between  A1  vs  W  A=B=fair  pos  correlations  OR 


Bias  between  A1  vs  W  A=B=good  pos  correlations  OR 


Figure  5-8  -  When  the  levels  of  performance  are  equal  between  systems  A0  and  Bv#e,  it  appears  that  positive 
correlation  negatively  impacts  bias.  For  reference,  P(rtl)  =  P(rt2)  =1/3  and  P(rnl)  =  P(l"n2)  =  2/3  under  the  across  I 

label  fusion  rule. 

As  seen  in  figure  5-8,  the  uncorrelated  curve  attains  the  highest  magnitude  of 
performance  bias,  but  drops  below  the  bias  curves  where  systems  Ae  and  BY;E  are 
positively  correlated  under  the  OR  rule.  What  is  even  more  unusual  is  what  occurs  when 
the  two  classification  systems  AH  and  BTjE  have  different  levels  of  performance. 
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Bias  between  A1  vs  W  A=fair  B=poor  pos  correlations  OR  Bias  between  A1  vs  W  A=poor  B=good  pos  corretations  OR 


Figure  5-9  -  When  systems  are  fused  with  different  levels  of  performance,  the  effect  of  positive  correlation  causes 
an  increase  in  bias  between  the  within  and  across  I  or  systems. 


If  the  performances  of  the  two  classification  systems  in  question  are  different,  it  is 
interesting  to  note  the  relative  lack  of  effect  on  positively  correlated  classification 
systems.  From  the  figure  above,  the  correlated  bias  curves  overlap  considerably  while  the 
uncorrelated  curve  is  clearly  influenced  by  the  change  in  combined  classification  system 
performance.  Note  again  that  the  maximum  magnitude  of  bias  seems  to  remain  on  par 
with  the  individual  classification  system  with  superior  performance. 

The  effects  of  altering  target  weight  for  across  I  target  types  was  also  considered 
for  those  ROC  curves  fused  under  the  OR  rule.  A  trend  similar  to  that  seen  under  the 
AND  rule  can  be  seen  in  figure  5-10. 
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Bias  between  A1  vs  W  A=poor  B=good  pos  correlations  unequal  pnors  OR 

07r 


Figure  5-10  -  In  the  figure  on  the  left,  the  uncorrelated  fused  classification  system  has  the  highest  level  of  bias 
when  both  systems  have  fair  performance.  On  the  right,  fusing  a  poor  A0  system  with  a  good  BY/E  system  in  a 
positively  correlated  environment  inflates  the  bias  when  positive  correlation  grows. 


By  altering  the  prior  probability  associated  with  one  of  the  targets  in  the  across  I 
combined  classification  system,  one  can  increase  the  performance  of  the  combined  across 
I  OR  ROC  curve  if  the  system  associated  with  the  target  that  has  the  majority  of 
probability  weight  has  superior  performance  (fig  5-10,  left).  As  was  seen  under  the  AND 
rule,  if  the  system  associated  with  the  increased  target  weight  has  inferior  performance, 
this  leads  to  increased  bias  between  within  and  across  I  OR  ROC  curves  (fig  5-10,  right). 

5. 2.1. 4  -  Effect  on  Bias  when  p  <0  under  the  OR  rule 
The  presence  of  negative  correlation  can  have  a  significant  impact  on  the 
magnitude  of  bias  between  within  and  across  I  label-fused  ROC  curves  under  the  OR 
rule.  Under  any  level  of  negative  correlation,  the  boost  to  performance  for  an  OR  ROC 
curve  is  considerably  greater  than  when  p  =  0.  If  the  performance  levels  are  varied,  this 
same  trend  continues. 
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Bias  between  A1  vs  W  A=B=good  neg  correlations  OR 


0.35 


Bias  between  A1  vs  W  A=poor  B=fair  neg  correlations  OR 


0  45r 


max  p+q-pq=r  (cl  =  c2  =  ;.  c3  =  x.  c4  =  solid) 


Bias  between  A1  vs  W  A=poor  B=fair  neg  correlations  unequal  priors  OR 


max  p*q-pq=r  (cl  =  -  c2  =  c3  =  x.  c4  =  solid) 


Figure  5-11  -  When  the  classification  systems  are  negatively  correlated  (bold,  dashed,  and  dotted  curves),  the  bias 
between  within  and  across  I  combined  classification  systems  under  the  OR  rule  decreases.  Altering  the  prior  target 
prevalence  of  either  tx  or  t2  can  exaggerate  these  levels  of  bias  (bottom). 


As  seen  in  figure  5-11,  when  the  individual  classification  systems  are  negatively 
correlated  and  fused  under  the  OR  rule,  the  magnitude  of  bias  between  within  and  across 
I  label-fused  ROC  curves  is  decreased.  This  trend  is  seen  when  performances  of 
individual  systems  are  different  and  when  across  target  weights  are  altered  as  well. 
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5.2.2  -  The  Effect  of  Correlation  on  Bias  between  Across  II  and  Within  Combined 
Classification  Systems 

5.2.2. 1  -  Effect  on  Bias  when  p>0  under  the  AND  rule 
In  an  across  II  environment,  classification  system  AH  is  tuned  to  target  type  I  and 
classification  system  ByE  is  tuned  to  all  elements  in  the  target  partition.  Given  that  system 
BT;£  classifies  the  same  set  of  targets  for  both  the  within  and  across  II  systems  leads  to 
interesting  results  for  the  performance  bias  between  across  II  versus  within  label-fused 
ROC  curves. 


Figure  5-12  -  The  magnitude  of  performance  bias  between  within  and  across  II  label-fused  ROC  curves  is 
considerably  smaller  than  that  seen  in  the  previous  case.  This  is  likely  caused  by  system  BY;£  being  tuned  to  same 

partitions  under  both  label  fusion  rules. 


Recall  from  theorem  4.2.3  that  when  the  values  of  p  and  q  are  the  same  for  both 
classification  systems,  the  bias  at  a  given  false  positive  value  is 

j^maxpq=r[T^  (q)(T]f  (p)  —  Tf1  (p))].  Hence,  the  bias  is  largely  a  function  of  the 

difference  in  performance  of  classification  system  AH  under  the  competing  label  fusion 
rules.  Under  the  AND  rule,  the  magnitude  of  bias  between  within  and  across  //combined 
label-fused  ROC  curves  is  considerably  smaller  than  that  observed  between  within  and 
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across  I  curves.  This  comparison  highlights  the  relative  dominance  of  classification 
systems  that  are  designed  to  label  all  target  elements  in  the  environment.  The  presence  of 
the  common  system  B,,x  produces  cases  where  the  level  of  performance  for  both  fused 
systems  is  quite  comparable.  The  across  II  label-fused  system  does  not  always  compete 
on  par  with  the  within  label- fused  system.  In  particular,  it  is  necessary  that  the 
performance  of  classification  system  BY;£  is  at  least  as  good  as  that  for  system  Ae.  In 
figure  5-12,  the  bias  curve  on  the  right  shows  the  output  for  all  tested  levels  of  correlation 
greater  than  or  equal  to  zero.  The  performance  bias  between  within  and  across  II  label- 
fused  ROC  curves  under  that  AND  rule  drops  to  zero  quickly  for  any  level  of  positive 
correlation  when  the  performance  of  system  Ae  is  equal  to  the  performance  of  By,e  (good 
performance  in  this  instance).  However,  if  the  performance  of  the  two  individual  systems 
is  not  the  same  as  seen  in  figure  5-13  (top  left),  the  magnitude  of  performance  bias 
between  the  two  label-fused  ROC  curves  reaches  levels  that  were  common  when 
comparing  within  and  across  I  fusion. 

Bias  between  A2  vs  W  A=good  B=poor  pos  correlations  Bias  between  A2  vs  W  A=fair  B=poor  pos  correlations 
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Bias  between  A2  vs  W  A=poor  B=good  pos  correlations 


Bias  between  A2  vs  W  A=poor  B=fair  pos  correlations 


Figure  5-13  -  As  demonstrated  in  the  case  of  the  unequal  weight  distribution  in  across  I,  if  system  BY/£  performs  at  a 
lower  level  than  system  A0,  the  ROC  curve  of  the  combined  system  is  "anchored"  by  the  performance  of  system 
BY/£.  Increasing  the  prior  target  prevalence  of  system  A0  (bottom  graphs)  further  decreases  the  bias  between  within 

and  across  II  and  ROC  curves. 

It  appears  that  as  the  performance  of  system  BY;£  decreases,  the  difference  in 
partitioning  of  events  for  classification  system  AH  under  the  two  competing  label  fusion 
rules  takes  precedence  in  determining  the  performance  bias  between  the  two  systems. 

It  is  important  to  note  that  under  the  AND  rule,  positive  correlation  between 
individual  classification  systems  appears  to  increase  the  magnitude  of  performance  bias 
between  label-fused  ROC  curves  when  system  B7;£  is  operating  at  an  equal  or  higher  level 
of  performance  than  system  A#.  This  is  unexpected  given  the  results  that  are  seen  in  the 
across  I  and  also  the  across  III  cases  where  under  the  AND  rule,  positive  correlation 
decreases  the  magnitude  of  performance  bias  between  within  and  across  systems 
regardless  of  performance  levels.  Though  difficult  to  see,  in  figures  5-12  and  5-13,  if 
system  B,,x  is  operating  at  an  equal  or  superior  level  of  performance,  the  uncorrelated 
ROC  curves  show  the  lowest  amount  of  performance  bias.  However,  this  does  make 
sense.  Though  the  performance  of  system  By>£  is  scaled  equivalently  for  both  within  and 
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across  II  label  fusion,  the  difference  in  performance  of  system  Ae  becomes  more  drastic 
between  the  two  label  fusion  rules,  leading  to  increased  performance  bias  between  the 
two  systems. 

5. 2. 2. 2  -  Effect  on  Bias  when  p  <0  under  the  AND  rule 
Negative  correlation  appears  to  have  small  and  unpredictable  effects  on  the 
magnitude  of  performance  bias  between  label-fused  within  and  across  II  ROC  curves. 
This  parallels  quite  nicely  with  the  effects  that  were  documented  when  investigating  the 
same  scenario  for  across  I  and  within  label-fused  ROC  curves  under  the  AND  rule. 


Bias  between  A2  vs  W  A=B=poor  neg  corretations  Bias  between  A2vs  W  A=good  B=poor  neg  correlations 


Figure  5-14  -  Negative  correlation  has  little  to  no  effect  on  the  bias  between  within  and  across  II  label-fused 
systems  under  the  AND  rule.  For  bottom  pictures,  P(rtl)  =1/2  and  P(rt)  =  2/3. 
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Figure  5-14  highlights  four  examples  of  the  effects  (or  lack  thereof)  on  bias  when 
altering  certain  properties  of  the  environment.  On  the  left  hand  side,  when  both  individual 
classification  systems  are  operating  at  a  poor  performance  level,  the  uncorrelated  curve  is 
well  above  the  negatively  correlated  curves.  Notice  that  on  the  bottom,  increasing  the 
likelihood  of  a  ti  event  reduces  the  magnitude  of  bias  in  both  situations.  It  is  more  clearly 
pronounced  when  altering  the  performance  of  systems  Ae  and  ByE. 

5. 2. 2. 3  -  Effect  on  Bias  when  p>0  under  the  OR  rule 
When  comparing  label-fused  ROC  curves  for  the  within  and  across  I  systems 
under  the  OR  rule,  it  was  noted  that  positive  correlation  had  small  and  unusual  effects  and 
negative  correlation  produces  a  clear  trend  of  decreasing  performance  bias.  This  same 
general  trend  is  seen  again  here. 

In  figure  5-15,  the  focus  is  on  those  instances  where  the  levels  of  performance  of 
individual  systems  Ae  and  Byx  is  different.  It  is  seen  that  when  system  By;:  has 
considerably  superior  performance,  the  effect  of  positive  correlation  is  hardly  noticeable 
as  the  maximum  magnitude  never  exceeds  0.045  tpr.  On  the  right  hand  side,  switching 
the  performance  of  the  systems  inflates  the  bias.  Hence,  it  is  seen  again  that  as  long  as 
system  Byx  is  performing  at  an  equal  or  superior  level  of  performance,  positive 
correlation  decreases  bias  and  the  opposite  is  true  when  the  roles  are  reversed. 
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Bias  between  A2  vs  W  A=poor  B=good  pos  correlations  OR  Bias  between  A2  vs  W  A-good  B=poor  pos  correlations  OR 


Figure  5-15  -  Notice  the  varying  magnitudes  of  bias  when  classification  system  BY/£  is  respectively  good  and  poor.  If 
BY/£  has  good  performance,  the  performance  of  system  A0  hardly  matters  (max  bias  of  approximately  0.04). 
Conversely,  if  the  performance  of  BY#£  is  poor,  the  bias  between  within  and  across  II  classifications  systems  can  be 
considerable.  Finally,  on  the  bottom  is  the  bias  curve  when  both  systems  perform  at  the  fair  level. 

5. 2. 2. 4  Effect  on  Bias  when  p  <0  under  the  OR  rule 
Given  the  results  that  were  seen  in  section  5.2. 1.4,  it  is  anticipated  that  in  the 
presence  of  negative  correlation,  the  bias  between  within  and  across  II  combined  systems 
will  decrease  when  the  correlation  coefficient  is  negative.  Indeed,  this  is  the  case. 
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Bias  between  A2  vs  W  A=B=good  neg  correlations  OR 


0.35 


Bias  between  A2  vs  W  A=good  B=poor  neg  correlations  OR 


0.2  0.3  0.4  0.5  0.6  07  08  09  1 

max  p+q-pq=r  (cl  =  c2  =  :,  c3  =  x,  c4  =  solid) 


Figure  5-16  -  When  p  <  0  the  magnitude  of  bias  between  within  and  across  II  label-fused  ROC  curves  under  the  OR 

rule  decreases. 


It  was  demonstrated  in  ( Won,  Cho:  2003 )  that  correlation  can  impact  the 
performance  of  classification  systems.  Although  that  particular  document  was  not 
concerned  with  label  fusion,  it  is  seen  here  that  the  boost  to  performance  of  the  fused 
systems  is  causing  the  two  ROC  curves  to  become  closer  in  value.  Some  of  this  bias  may 
appear  artificial  as  this  perceived  boost  is  being  dramatized  by  setting  a  maximum  true 
positive  rate  of  1,  however,  if  the  two  curves  are  permitted  to  break  the  measure  of  the 
set,  it  can  be  seen  that  the  relative  difference  in  performance  is  indeed  decreasing.  The 
sharp  decline  in  bias  seen  in  the  bias  curves  under  the  OR  rule  is  caused  by  the  within 
label-fused  system  reaching  its  max  true  positive  rate  and  remaining  constant  as  the 
across  II  label-fused  ROC  curve  converges  to  the  same  maximum  true  positive  rate. 

As  seen  in  figures  5-16  and  5-17,  the  level  of  performance  of  system  Byx  has  the 
most  dramatic  effect  on  magnitude  of  bias  between  the  two  label-fused  ROC  curves.  This 
magnitude  is  cut  nearly  in  half  when  increasing  the  probability  of  observing  a  ti  event  for 
system  Ag. 
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Bias  between  A2  vs  W  A=fair  B=good  neg  correlations  unequal  priors  OR  Bias  between  A2  vs  W  A=good  B=poor  neg  correlations  unequal  priors  OR 


Figure  5-17  -  Increasing  the  prevalence  of  tx  in  an  across  II  label-fused  system  decreases  the  bias.  In  this  instance, 

P(rtl)  =  1/2  and  P(rt)  =  2/3. 

5.2.3  -  The  Effect  of  Correlation  on  Bias  between  Across  III  and  Within 
Combined  Classification  Systems 

The  effect  of  correlation  on  the  bias  between  across  III  label-fused  systems  and 
within  label-fused  classification  systems  shares  many  similarities  with  the  bias  between 
within  and  across  I  systems.  This  largely  has  to  do  with  the  fact  that  the  target  sets  for 
classification  systems  Ae  and  have  one  mutually  exclusive  target  type  (ti  and  t3  for 
classification  system  AH  and  BT;£,  respectively)  and  both  also  label  the  same  subset  of 
targets,  t2.  This  decreases  the  bias  between  label-fused  ROC  curves  as  compared  to 
across  I  where  the  target  sets  for  both  classification  systems  were  mutually  exclusive. 

5.2.3. 1  -  Effect  on  Bias  when  p>0  under  the  AND  rule 

Consider  the  event  where  the  prior  probability  of  observing  a  target  event  is  equal 
for  classification  systems  Ae  and  BY;£  under  the  across  III  label  fusion  rule.  In  the  case 
that  two  classification  systems  are  positively  correlated  under  both  the  within  and  across 
III  label  rule,  it  is  seen  that  the  magnitude  of  bias  decreases.  This  is  to  be  expected  given 
the  similarity  to  the  across  I  scenario 
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Bias  between  A3  vs  W  A=B=fair  pos  correlations  Bias  between  A3  vs  W  A=B=good  pos  correlations 


Figure  5-18  -  When  p  >  0  (bold,  dashed,  and  dotted  curves),  the  bias  between  combined  within  and  across  III 

systems  under  the  AND  rule  is  decreased. 


When  the  correlation  coefficient  takes  on  values  0.3  and  0.5  (dashed  and  dotted 
curves  respectively  in  figure  5-18),  the  reduction  in  performance  bias  is  minimal 
compared  to  the  drastic  case  when  the  coefficient  takes  the  value  0.8  (bold  x  curve).  The 
change  in  performance  bias  is  muted  in  some  respects  in  this  scenario  likely  because  of 
the  presence  of  the  common  t2  subset  for  both  systems.  As  noted  earlier,  the  magnitude  of 
bias  appears  to  be  most  closely  related  to  the  classification  system  with  the  superior  level 
of  performance.  This  is  a  trend  that  has  persisted  through  all  tested  scenarios  and  appears 
again  in  figure  5-19  where  the  levels  of  performance  of  individual  systems  was  altered. 
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Bias  between  A3  vs  W  A=poor  B=good  pos  correlations  Bias  between  A3  vs  W  A=good  B=fair  pos  correlations 


Figure  5-19  -  Combining  two  individual  systems  of  different  performance  under  the  across  III  and  within  label 

fusion  rules. 


Consider  altering  the  target  weight  of  observing  a  t n  event  to  P( t12)  =  —  and  the 


3 

probability  of  observing  a  t23  event  to  P(t23)  =  —  for  the  across  III  combined 
classification  system  under  the  AND  rule. 


Bias  between  A3  vs  W  A=B=good  pos  correlation  unequal  priors  Bias  between  A3  vs  W  A=poor  B=good  pos  correlation  unequal  priors 


Figure  5-20  -  The  bias  decreases  when  the  individual  classification  system  with  the  majority  of  prior  target 
prevalence  has  a  higher  or  equivalent  level  of  performance.  The  opposite  is  true  when  the  system  with  the  majority 

of  target  prevalence  has  poorer  performance. 
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Not  surprisingly,  the  bias  curves  output  when  quantifying  the  bias  between  across 
III  and  within  ROC  curves  strongly  resemble  the  results  of  section  5.2.1 
5.23.2  -  Effect  on  Bias  when  p  <0  under  the  AND  rule 
When  the  two  individual  classification  systems  are  negatively  correlated  under  the 
AND  rule,  the  bias  curves  overlap  considerably.  It  is  unclear  why  negative  correlation 
seems  to  affect  both  within  and  across  III  fused  systems  equally  (that  is,  there  is  no 
difference  in  performance  for  the  within  and  across  III  label-fused  ROC  curves). 

Bias  between  A3  vs  W  A=B=fair  neg  correlations  Bias  between  A3  vs  W  Affair  B=good  neg  correlations 


Figure  5-21  -  Negative  correlation  has  little  effect  on  combined  AND  classification  systems. 


Altering  prior  target  prevalence  can  be  beneficial  to  decreasing  the  difference  in 
responses  between  within  and  across  III  systems  when  the  system  responsible  for 
classifying  the  majority  of  elements  performs  at  a  level  equal  to  or  superior  to  the  other 
system.  The  converse  is  true  when  the  system  tasked  with  classifying  the  majority  of 
elements  in  the  environment  has  inferior  performance. 
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Bias  between  A3  vs  W  A=B=fair  neg  correlations  unequal  priors 


Bias  between  A3  vs  W  A=fair  B=good  neg  correlations  unequal  priors 


Figure  5-22  -  Oddly,  combining  systems  of  different  performance  seems  to  eliminate  fluctuations  in  bias  under  the 

AND  rule. 


5.2.33  -  Effect  on  Bias  when  p>0  under  the  OR  rule 
Under  the  OR  rule,  if  the  two  individual  classification  systems  are  positively 
correlated,  then  the  closer  the  two  systems  are  to  being  independent,  the  less  performance 
bias  exists.  This  makes  sense  as  this  same  trend  occurred  when  investigating  the 
performance  bias  between  across  I  and  within  label-fused  ROC  curves  under  the  OR  rule 
when  the  correlation  coefficient  was  non-negative. 

Bias  between  A3  vs  W  A=8=good  pos  correlations  OR  Bias  between  A3  vs  W  A=poor  B=good  pos  correlations  OR 


Figure  5-23  -  When  the  two  classification  systems  are  independent,  the  bias  between  across  III  and  within  label- 
fused  ROC  curves  is  minimized  under  the  OR  rule  and  the  correlation  coefficient  is  non-negative. 
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Not  depicted  here  is  the  altering  of  individual  target  prevalence  for  across  III 
target  types.  The  trend  is  the  same  as  that  seen  in  the  previous  two  subsections  and  will 
not  be  discussed  here  to  avoid  redundancy. 

5. 2. 3. 4  -  Effect  on  Bias  when  p  <0  under  the  OR  rule 
When  the  correlation  constant  is  negative,  the  results  are  what  one  would  expect: 
negative  correlation  under  the  OR  rule  implies  decreased  bias  between  within  and  across 
III  combined  classification  systems.  The  magnitude  of  performance  bias  between  within 
and  across  III  systems  is  less  than  that  observed  in  the  across  I  scenario  due  to  the 
common  t2  subset. 

Bias  between  A3  vs  W  A=poor  B=fair  nag  correlations  OR  Bias  between  A3  vs  W  A=poor  B=good  neg  correlations  unequal  priors  OR 


Figure  5-24  -  Negative  correlation  leads  to  "improvement"  in  the  performance  of  the  combined  OR  ROC  curves. 

5.3  -  Altered  Target  Environments  and  its  Effects  on  Bias 

Recall  that  there  are  three  different  types  of  target  environments  being 
investigated  in  this  simulation.  The  first  target  environment,  the  target  enhanced 
environment,  has  been  expounded  upon  in  the  section  above.  In  a  target  enhanced 
environment,  the  proportion  of  targets  to  non-targets  is  2/3  to  1/3.  The  other  two  target 
environments,  the  target  rich  and  target  deficient  environments  will  be  summarized  here. 
Recall  that  in  a  target  rich  environment,  the  proportion  of  targets  is  4/5  to  1/5  and  in  a 
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target  deficient  environment,  the  proportion  of  targets  to  non-targets  is  1/5  to  4/5.  Note 
that  only  the  overall  probability  of  observing  a  target  or  non-target  event  is  being  changed 
here. 

5.3.1  -  Altered  Target  Environment  Across  I  versus  Within 

Depending  on  the  type  of  target  environment,  the  resulting  difference  in  bias  can 
be  quite  drastic.  In  a  target  rich  environment,  the  bias  between  within  and  across  I 
classification  systems  under  the  AND  rule  is  increased  marginally.  The  type  of 
environment  does  not  appear  to  change  the  trends  that  were  seen  in  the  target  enhanced 
environment,  though  the  magnitude  of  bias  may  be  affected.  This  is  most  notably  seen  in 
the  target  deficient  environment.  It  appears  that  reducing  probability  of  observing  a  target 
event  in  an  environment  has  the  greatest  effect  on  bias  between  within  and  across  I 
combined  classification  systems  (figure  5-25  middle  row).  In  the  target  rich  environment, 
the  shape  of  the  bias  curve  resembles  that  of  the  target  enhanced  environment,  but  notice 
that  the  curve  appears  stretched  over  the  x-axis.  This  occurs  as  the  false  positive  rate  at 
which  these  levels  of  performance  are  being  registered  is  increasing.  That  is  to  say,  the 
higher  the  probability  of  observing  a  target  event,  the  more  likely  it  becomes  that  the 
individual  classification  systems  incorrectly  label  targets  with  the  non-target  label. 
Therefore,  it  makes  logical  sense  in  an  environment  with  proportionally  few  target  events 
that  classification  systems  with  good  performance  would  correctly  label  the  true  target 
events  with  the  target  label  very  effectively.  That  is  to  say,  given  that  it  is  unlikely  to 
observe  a  target  event,  the  probability  that  a  target  is  mislabeled  is  proportionally  smaller. 
Even  at  a  modest  positive  correlation  of  0.3  (dashed  line),  the  bias  between  across  I  and 
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within  classification  systems  drops  off  tremendously  under  the  AND  rule  (figure  5-25). 
Increasing  this  constant  only  drives  the  bias  to  zero  at  a  faster  rate. 


Bias  between  A1  vs  W  A=B=good  pos  correlation  rich 


Bias  between  A1  vs  W  A=fair  B=good  pos  correlation  rich 


Bias  between  A1  vs  W  A=B=good  pos  correlation  def 


Bias  between  A1  vs  W  A=fair  B=good  pos  correlation  def 


Bias  between  A1  vs  W  A=B=good  pos  correlations 


Bias  between  A1  vs  W  Affair  B=good  pos  correlations 


Figure  5-25  -  Altering  the  distribution  of  targets  and  non-targets  across  the  entire  event  set  has  huge  implications 
for  the  performance  of  combined  systems.  In  a  target  deficient  environment,  the  bias  between  within  and  across 
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systems  approaches  zero  in  the  presence  of  positive  correlation  under  the  AND  rule.  In  a  target  rich  environment, 


the  overall  bias  increases  between  within  and  across  classification  systems. 

Altering  the  prior  target  prevalence  for  across  type  targets  in  different 
environments  for  systems  combined  under  the  AND  rule  also  produces  similar  results  as 
were  seen  in  section  5.2  (figure  5-26).  Negatively  correlated  classification  systems  under 
the  AND  rule  and  positively  correlated  classification  systems  under  the  OR  rule  will  not 
be  discussed  here.  If  the  reader  wishes  to  visually  inspect  these  cases,  he  or  she  is 
directed  to  the  contents  of  appendix  B. 

Bias  between  A1  vs  W  A=poor  B=good  pos  correlation  unequal  priors  Bias  between  A1  vs  W  A=poor  B=good  pos  correlation  def  unequal  priors 


Bias  between  A1  vs  W  A=poor  B=good  pos  correlation  rich  unequal  priors 


Figure  5-26 -Altering  individual  target  prevalence  produces  similar  results  regardless  of  target  environment. 
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Consider  the  label-fused  ROC  curves  under  the  OR  rule  in  these  altered 


environments.  As  suggested  before,  by  reducing  the  probability  of  a  target  event 
occurring,  the  relative  performance  of  within  and  across  label-fused  classification 
systems  at  correctly  labeling  their  respective  target  types  should  increase.  Furthermore,  if 
the  two  classification  systems  are  negatively  correlated  under  the  OR  rule,  this  boost  to 
performance  can  be  dramatic  (figure  5-27,  middle  row). 


Bias  between  A1  vs  W  A=fair  B=good  neg  correlations  OR 


Bias  between  A1  vs  W  A=B=poor  neg  correlations  OR 


Bias  between  A1  vs  W  A=fair  B=good  neg  correlation  def  OR 
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Bias  between  Ai  vs  W  A=fair  B=good  neg  cofretation  rich  OR  Bias  between  A1  vs  W  A=B=poor  neg  correction  rich  OR 


Figure  5-27  -  As  was  observed  under  the  AND  rule,  the  relationship  between  performance  bias  and  probability  of 
observing  a  target  event  in  the  environment  in  inversely  proportional.  That  is  to  say,  by  increasing  the  probability 
of  observing  a  target  event,  the  individual  classification  systems  are  more  likely  to  incorrectly  label  targets  with  a 

non-target  label. 


As  there  are  no  clear  trends  that  can  be  analyzed  when  two  classification  systems 
are  positively  correlated  under  the  OR  rule,  these  results  will  not  be  discussed  here.  The 
bias  curves  are  included  in  appendix  B  if  the  reader  wishes  to  observe  the  output  for  him 
or  herself. 

5.3.2  -  Altered  Target  Environments  Across  II  versus  Within 
Altering  the  target  environment  when  comparing  across  II  label-fused  ROC 
curves  to  within  label-fused  ROC  curves  seems  to  dramatize  the  results  from  section  5.2. 
Decreasing  the  number  of  targets  in  the  event  set  causes  the  bias  between  the  systems  to 
decrease  more  sharply  and  increasing  the  number  of  targets  in  the  environment  increases 
the  bias  between  systems.  Altering  the  ratio  of  fi’s  for  classification  system  A@  has  the 
same  effects  as  before,  but  is  not  pictured  here.  These  figures  have  been  included  in 
appendix  B. 
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Bias  between  A2  vs  W  A=B=fair  pos  correlations 


Bias  between  A2  vs  W  A=fair  B=poor  pos  correlations 


Bias  between  A2  vs  W  A=fair  B=poor  pos  correlation  def 


025 


Bias  between  A2  vs  W  A=fair  B=poor  pos  correlation  rich 


max  pq=r  (c4  =  sold.  c5  =  --.c6=  c7  =  x) 


Figure  5-28  -  Decreasing  the  proportion  of  targets  in  the  environment  greatly  affects  bias  between  within  and 

across  II  label-fused  systems  under  the  AND  rule. 


As  seen  above,  the  proportion  of  target  in  the  environment  is  inversely 
proportional  to  the  magnitude  of  bias  that  occurs  between  within  and  across  //label-fused 
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ROC  curves  under  the  OR  rule  when  the  correlation  coefficient  is  negative.  Positive 
correlation  under  the  OR  rule  is  not  discussed  here,  but  this  can  be  investigated  in 
appendix  B. 


Bias  between  A2  vs  W  A=B=poor  neg  correlations  OR 


Bias  between  A2  vs  W  A=B=poor  neg  correlation  def  OR 


Bias  between  A2  vs  W  A=B=poor  neg  correlation  rich  OR 


Figure  5-29  -  Altering  the  target  environment  when  quantifying  the  performance  bias  between  across  II  and  within 

label-fused  ROC  curves  under  the  OR  rule. 


5.3.3  -  Altered  Target  Environments  Across  III  versus  Within 
Given  the  similarities  between  across  I  and  across  III  label  fusion,  the  results  of 
the  simulation  are  predictable.  The  presence  of  the  common  target  subset,  t2,  reduces  the 
bias  between  across  III  and  within  label-fused  systems  regardless  of  which  Boolean  rule 
is  being  utilized.  Once  again,  all  figures  not  discussed  (negative  correlation  under  the 
AND  rule,  positive  correlation  under  the  OR  rule,  and  altering  prior  target  prevalence  for 
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individual  classification  systems  under  the  across  III  label  fusion  rule)  here  are  included 
in  appendix  B. 


Bias  between  A3  vs  W  A=B=fair  pos  correlations 


Bias  between  A3  vs  W  A=good  B=fair  pos  correlations 


Bias  between  A3  vs  W  A=B=fair  pos  correlation  def 


Bias  between  A3  vs  W  A=good  B=fair  pos  correlation  def 


Bias  between  A3  vs  W  A=B=fair  pos  correlation  rich 


Bias  between  A3  vs  W  A=good  B=fair  pos  correlation  rich 


Figure  5-30  -  The  presence  of  a  shared  subset  seems  to  reduce  the  bias  between  across  III  and  within  ROC  curves 

under  the  AND  rule. 
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In  the  case  of  the  OR  rule,  the  alteration  of  target  environment  produces  similar  results. 


Bias  between  A3  vs  W  A=poor  B=good  neg  correlations  OR 

0.35  r 

0.3  - 


Bias  between  A3  vs  W  A=poor  B=good  neg  correlation  def  OR 

0.35  r 


0  3 


Bias  between  A3  vs  W  A=poor  B=good  neg  correlation  rich  OR 


Figure  5-31  -  Altering  target  environment  causes  similar  changes  under  the  OR  rule. 

In  summary,  all  of  the  evidence  points  to  a  few  clear  things.  Given  the 
assumptions  made  in  this  simulation,  the  within  combined  classification  system 
outperforms  any  across  label- fused  system  in  terms  of  pure  performance.  The  across 
systems  approaches  the  performance  of  the  within  system  as  the  overlap  in  target 
classification  grows.  The  performance  bias  can  be  tweaked  as  was  demonstrated  here. 
The  most  effective  way  to  approximate  these  two  ROC  curves  seems  to  be  altering  the 
probability  of  observing  a  target  event  either  for  across  type  targets  or  the  proportion  of 
targets  in  the  population  event  set.  Though  clearly  these  cannot  be  changed  in  the  field, 


128 


these  are  variable  that  the  experimenter  can  use  to  alter  the  performance  of  legacy 
systems.  Given  that  some  trends  were  seen  for  the  AND  and  OR  rules,  increasing  or 
decreasing  the  correlation  coefficient  also  gives  the  experimenter  tools  for  adjusting  the 
performance  of  legacy  systems. 
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VI  Discussion 


In  this  thesis,  a  method  for  determining  the  bias  between  different  label-fused 
classification  systems  was  developed.  This  notion  of  bias  between  label-fused 
classification  systems  carries  its  own  set  of  pros  and  cons.  First,  the  results  of  this  thesis 
bring  to  light  questions  regarding  the  importance  of  pre-existing  knowledge  of  those 
elements  to  be  classified.  Regardless  of  the  type  of  Boolean  rule,  target  environment,  or 
prior  target  weighting,  within  label-fusion  consistently  outperforms  any  type  of  across 
label  fusion.  In  the  context  of  raw  performance,  it  seems  clear  that  one  would  choose  to 
use  within  label-fusion.  If  the  features  of  the  elements  of  interest  are  not  partitioned  in 
such  a  way  that  the  assumptions  of  within  label  fusion  can  be  me,  then  there  is  the 
possibility  that  incorrectly  applying  this  assumption  overestimates  system  performance. 
As  was  demonstrated  in  the  simulation,  the  difference  in  response  between  within  and 
across  label  fusion  could  quite  substantial  and  incorrectly  applying  the  wrong  type  of 
fusion  can  lead  to  errors  in  reported  performance. 

That  being  said,  there  are  ways  to  minimize  the  bias  between  the  two  systems.  In 
across  I  and  across  III  fused  systems,  if  the  prior  target  weight  associated  with  the 
superior  individual  classification  system  (either  system  A  or  B)  is  increased,  then  the  bias 
between  within  and  across  systems  is  decreased.  Likewise,  it  was  seen  under  the  AND 
rule  that  increasing  the  correlation  coefficient,  p,  can  also  assist  in  decreasing  bias 
between  the  systems.  Analogously,  under  the  OR  rule,  decreasing  p  reduced  the  bias 
between  the  two  types  of  label-fused  systems. 
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Recall  that  under  most  circumstances,  target  prevalence  cannot  be  altered  and  pre¬ 
existing  knowledge  of  the  partition  of  the  event  set  may  be  unknown.  However, 
knowledge  of  a  new  target  environment  and  the  systems  that  are  being  fused(what  the 
systems  are  tuned  to  detect),  may  provide  the  end-user  with  the  means  to  appropriately 
adjust  the  performance  and  optimal  thresholds  for  performance  by  fusing  legacy  systems. 
Such  ability  reflects  a  direct  application  of  flexible  engineering. 

A  limitation  of  using  a  set  of  empirical  ROC  curves  is  that  a  continuum  of  bias 
values  will  not  always  be  available  for  computation  (i.e.  there  may  be  values  for  the  fused 
false  positive  “r”  that  are  unique  to  each  label-fused  ROC  curve,  meaning  not  all  bias 
values  may  be  computed).  Small  departures  from  pure  independence,  that  is,  small  values 
of  correlation,  may  not  alter  the  ROC  curve  of  the  fused  system  significantly.  In  this 
thesis,  there  were  two  cases  where  independence  was  crucial  to  theoretical  and  applicable 
development:  1)  Individual  classification  systems  AH  and  EC  must  be  independent.  2)  the 
classification  systems  must  be  conditionally  independent  with  respect  to  the  non-target 
partition  of  the  event  set.  Much  work  has  been  done  with  respect  to  the  first  assumption 
and  many  researchers  have  been  able  to  create  independent  classification  systems. 
Furthermore,  the  cost  functions  derived  by  (Schubert:  2005 )  make  no  assumption  about 
the  independence  of  classification  systems  Ae  and  EC.  Rather,  the  simplification  of  the 
cost  function  when  the  correlation  constant  p  =  0  represents  the  formula  when  the  two 
systems  to  be  fused  are  independent.  The  second  assumption  is  nested  within  the  same 
equation;  namely  that  the  false  positive  for  the  label-fused  system  A  and  B  under  the 
AND  rule  is  indeed  pq  (or  p+q-pq  under  the  OR  rule)  At  times,  this  may  be  a  strong 
assumption  to  make  about  the  the  classification  systems  and  the  non-target  partition  of 
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the  event  set.  As  a  result,  this  assumption  has  to  be  built  into  any  classification  system 
that  would  use  this  approach.  Future  work  may  examine  this  independence  and  determine 
how  sensitive  the  label-fused  ROC  curves  are  to  this  assumption.  It  is  not  out  of  the 
realm  of  possibility  that  this  assumption  can  be  made,  but  to  do  so  without  closely 
considering  the  situation  can  lead  to  erroneous  results. 

Arguably  the  most  important  part  of  this  thesis  was  the  development  of  the 
formulas  that  quantify  the  performance  bias  between  the  different  types  of  label-fusion. 
Given  that  these  equations  are  built  from  the  cost  functions  themselves,  they  include  all 
the  variables  necessary  to  make  the  transformation  from  one  label-fused  ROC  curve  to 
another  other.  Secondly,  these  equations  are  built  from  the  individual  classification 
systems  A@  and  EC;  meaning  this  is  ample  information  to  fuse  the  classification  systems 
and  determine  the  difference  in  performance  between  them.  Hence,  given  knowledge 
about  systems  Ae  and  EC,  it  is  possible  to  pinpoint  those  variables  that  are  causing  the 
bias  and  alter  them  accordingly.  It  is  assumed  that  if  label-fusion  is  going  to  be  used,  one 
has  to  make  an  assumption  regarding  the  partitioning  of  the  event  set.  Regardless  of  what 
this  original  assumption  may  be,  as  more  knowledge  becomes  available  about  the  true 
partitions,  these  variables  can  be  tuned  to  model  the  dynamic  truth.  In  such  a  manner, 
determining  the  performance  of  the  combined  classification  system  is  no  longer  a  barrier 
when  developing  such  flexibility  in  system  design. 

In  order  to  use  these  formulas,  a  hard  value  for  the  prior  probability  of  target  and 
non-target  events  is  needed.  Choosing  only  one  of  a  continuum  of  values  may  introduce 
errors,  but  the  use  of  training  (such  as  with  neural  networks)  may  mitigate  these  potential 
issues. 
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Should  values  be  available,  one  benefit  of  having  the  simple  formulas  presented 
herein  is  that  it  is  very  easy  to  apply  computationally.  The  algorithm  provided  in 
appendix  C  requires  only  finding  the  set  intersection  of  false  positive  values  and 
subtracting  the  corresponding  true  positive  values  to  compute  the  bias  and  adjust 
performance  between  within  and  across  label-fused  ROC  curves.  Another  benefit 
associated  with  these  formulas  is  that  they  can  be  easily  applied  to  legacy  data.  Given  the 
tenuous  future  of  research  funding,  the  ability  to  apply  new  ideas  to  pre-existing  scientific 
data  (some  of  which  have  already  made  assumptions  regarding  the  distribution  of  target 
and  non-target  features),  these  functions  will  be  easily  adaptable.  Further,  these  formulas 
also  support  system  flexibility  so  that  different  combinations  of  individual  classification 
systems  may  be  combined  together  and  performance  appropriately  determined  from  only 
the  information  of  the  performances  (ROC  curves)  of  the  individual  systems. 

6. 1  Future  Work 

It  is  clear  that  there  are  plenty  of  questions  regarding  this  approach  to  fusion  that 
have  yet  to  be  answered.  First,  the  development  of  a  distribution  for  the  prior  probability 
of  targets  and  non-targets  appears  pressing.  In  this  way,  the  weighting  associated  with 
individual  target  types  can  be  adjusted  dynamically.  Assuming  values  for  the  prior 
probability  of  targets  and  non-targets  in  an  environment  is  perfectly  suited  to  theory,  but 
in  practice,  building  a  system  this  specific  is  highly  inefficient.  Second,  developing  a 
method  that  eases  the  need  for  conditional  independence  of  classification  systems  with 
respect  to  the  non-target  event  set  could  have  interesting  ramifications.  It  is  common  in 
research  to  suggest  that  theoretical  classification  systems  are  indeed  “independent,”  but 
little  work  has  been  done  showing  what  effects  this  has  in  application.  Finally,  it  is  of 
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interest  to  extend  the  concepts  developed  here  to  any  type  of  ROC  curve.  Remember  that 
the  bias  formulas  currently  only  work  for  the  Boolean  AND  and  Boolean  OR  rule.  It 
would  be  interesting  to  extend  this  theory  to  different  classification  methods  such  as 
neural  networks  or  at  least  different  performance  measures. 
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Appendix  A 


In  this  appendix  you  will  find  all  of  the  ROC  curves  that  were  created  during  the 
simulation.  As  a  reference,  in  each  figure,  the  ROC  curves  of  the  specified  combined 
across  and  within  classification  system  are  plotted  along  with  the  ROC  curves  for 
individual  systems  Ae  and  Byx.  The  material  will  be  presented  in  the  following  order: 

1)  Combined  systems  in  the  enhanced  target  prevalence  environment 

a)  Combined  AND  ROC  curves 

b)  Combined  OR  ROC  curves 

c)  Combined  AND  ROC  curves  with  unequal  prior  target  weighting 

d)  Combined  OR  ROC  curves  with  unequal  prior  target  weighting 

2)  Combined  systems  in  the  rich  target  prevalence  environment 

a)  Combined  AND  ROC  curves 

b)  Combined  OR  ROC  curves 

c)  Combined  AND  ROC  curves  with  unequal  prior  target  weighting 

d)  Combined  OR  ROC  curves  with  unequal  prior  target  weighting 

3)  Combined  systems  in  the  deficient  target  prevalence  environment 

a)  Combined  AND  ROC  curves 

b)  Combined  OR  ROC  curves 

c)  Combined  AND  ROC  curves  with  unequal  prior  target  weighting 

d)  Combined  OR  ROC  curves  with  unequal  prior  target  weighting 
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A. 2  -  Rich  Target  Prevalence  Figures 
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A  ■  mM.  B  ■  dMh 


A =good  B=fair  AND  A3  vs  Within 


A=fair  B=poor  AND  A3  vs  Within 


A=B  =f air  AND  A3  vs  Within 


A =good  \i=poor  AND  A3  vs  Within 


A  =poor  B  =good  AND  A3  vs  Within 


A=B  =poor  AND  A3  vs  Within 


0  0.1  02  03  04  03  06  0.7  06  09  1 

A  ■  mM  B  -  dash 


N=fair  B  =good  AND  A3  vs  Within 


A  =poor  B=fair  AND  A3  vs  Within 
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02 


03  04 


03  04 


0.3  04 


A=B  =good  AND  unequal  priors  A1  vs 
Within 


A=B  =fair  AND  unequal  priors  A1  vs 
Within 


A=B  =poor  AND  unequal  priors  A1  vs 
Within 


A =good  B=fair  AND  unequal  priors  A1 
vs  Within 


A =good  B=poor  AND  unequal  priors 
A1  vs  Within 


Affair  B  =good  AND  unequal  priors  A1 
vs  Within 


A =fair  B=poor  AND  unequal  priors  A1 
vs  Within 


A =poor  B=good  AND  unequal  priors 
A1  vs  Within 


A =poor  B=fair  AND  unequal  priors  A1 
vs  Within 
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A=B  =good  AND  unequal  priors  A2  vs 
Within 


0  01  02  03  04  05  06  07  06  09 


A =good  B=fair  AND  unequal  priors  A2 
vs  Within 


A =fair  B=poor  AND  unequal  priors  A2 
vs  Within 


A=B  =fair  AND  unequal  priors  A2  vs 
Within 


A =good  B=poor  AND  unequal  priors 
A2  vs  Within 


A =poor  B=good  AND  unequal  priors 
A2  vs  Within 


A=B  =poor  AND  unequal  priors  A2  vs 
Within 


0  01  02  03  04  05  06  07  06  09 


Affair  B  =good  AND  unequal  priors  A2 
vs  Within 


A =poor  B=fair  AND  unequal  priors  A2 
vs  Within 
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A=B  =good  AND  unequal  priors  A3  vs 
Within 


A=B  =fair  AND  unequal  priors  A3  vs 
Within 


A=B  =poor  AND  unequal  priors  A3  vs 
Within 


A =good  B=fair  AND  unequal  priors  A3 
vs  Within 


A =good  B=poor  AND  unequal  priors 
A3  vs  Within 


Affair  B  =good  AND  unequal  priors  A3 
vs  Within 


A =fair  B=poor  AND  unequal  priors  A3 
vs  Within 


A =poor  B=good  AND  unequal  priors 
A3  vs  Within 


A =poor  B=fair  AND  unequal  priors  A3 
vs  Within 
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A=fair  B=poor  OR  A1  vs  Within 


A  =poor  B=good  OR  A1  vs  Within 


A  =poor  B=fair  OR  A1  vs  Within 


166 


oa 

/ 

?o.  / 


A=B  =good  OR  A2  vs  Within 


0  01  02  03  04  05  06  07  06  09 


A =good  B=fair  OR  A2  vs  Within 


A=fair  B=poor  OR  A2  vs  Within 


A=B  =fair  OR  A2  vs  Within 


A =good  B=poor  OR  A2  vs  Within 


A  =poor  B=good  OR  A2  vs  Within 


A=B  =poor  OR  A2  vs  Within 


st 

0.2  I 


0  01  02  03  04  05  06  07  06  09 


A  =fair  JS=good  OR  A2  vs  Within 


A  =poor  B=fair  OR  A2  vs  Within 
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ioi  / 
S05 


0  01  02  03  04  05  06  07  06  09  1 


A=B  =good  OR  A3  vs  Within 


A =good  B=fair  OR  A3  vs  Within 


A=fair  B=poor  OR  A3  vs  Within 


A=B  =fair  OR  A3  vs  Within 


A =good  B=poor  OR  A3  vs  Within 


A  =poor  B=good  OR  A3  vs  Within 


A=B  =poor  OR  A3  vs  Within 


0  0.1  02  03  04  03  06  07  06  09  1 

A  ■  mM  B  -  dnh 


A  =fair  JS=good  OR  A3  vs  Within 


A  =poor  B=fair  OR  A3  vs  Within 
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0  01  02  03  04  05  06  07  06  09  1 


A=B  =good  OR  unequal  priors  A1  vs 
Within 


0  01  02  03  04  05  06  07  06  09 


A =good  B=fair  OR  unequal  priors  A1 
vs  Within 


A =fair  B=poor  OR  unequal  priors  A1  vs 
Within 


A=B  =fair  OR  unequal  priors  A1  vs 
Within 


A =good  B=poor  OR  unequal  priors  A1 
vs  Within 


A =poor  B  =good  OR  unequal  priors  A1 
vs  Within 


A=B  =poor  OR  unequal  priors  A1  vs 
Within 


0  01  02  03  04  05  06  07  06  09 


Affair  B=good  OR  unequal  priors  A1 
vs  Within 


A =poor  B=fair  OR  unequal  priors  A1  vs 
Within 
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A=B  =good  OR  unequal  priors  A2  vs 
Within 


0  01  02  03  04  05  06  07  06  09 


A =good  B=fair  OR  unequal  priors  A2 
vs  Within 


A =fair  B=poor  OR  unequal  priors  A2  vs 
Within 


A=B  =fair  OR  unequal  priors  A2  vs 
Within 


A =good  B=poor  OR  unequal  priors  A2 
vs  Within 


A =poor  B  =good  OR  unequal  priors  A2 
vs  Within 


A=B  =poor  OR  unequal  priors  A2  vs 
Within 


0  01  02  03  04  05  06  07  06  09 


Affair  B=good  OR  unequal  priors  A2 
vs  Within 


A  vs  B  v*  Across  ■  A=P  B=F  vs  wtta  A =P  B=F  OP  target  defidert  environment  unequal  poors 


A =poor  B=fair  OR  unequal  priors  A2  vs 
Within 
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0  01  02  03  04  05  06  07  06  09  1 


A=B  =good  OR  unequal  priors  A3  vs 
Within 


A =good  B=fair  OR  unequal  priors  A3 
vs  Within 


A =fair  B=poor  OR  unequal  priors  A3  vs 
Within 


A=B  =fair  OR  unequal  priors  A3  vs 
Within 


A =good  B=poor  OR  unequal  priors  A3 
vs  Within 


A =poor  B=good  OR  unequal  priors  A3 
vs  Within 


A=B  =poor  OR  unequal  priors  A3  vs 
Within 


0  0.1  02  03  04  03  06  0.7  06  09  1 

A  ■  mM  B  -  dnh 


A =fair  B=good  OR  unequal  priors  A3 
vs  Within 


A =poor  B=fair  OR  unequal  priors  A3  vs 
Within 
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Appendix  B 


Included  in  this  appendix  are  all  of  the  bias  curves  generated  during  the 
simulation.  As  a  quick  outline,  in  each  figure  four  bias  curves  are  plotted:  the 
uncorrelated  curve  and  the  three  positively  or  negatively  correlated  curves  which  are 
specified  by  the  p  >  0  or  p  <  0  in  the  caption  below.  The  Boolean  rule  along  with  the 
distribution  of  prior  target  probability  is  also  included  in  the  captions.  The  figures  will  be 
presented  in  the  following  order: 

1)  Enhanced  Target  Prevalence 

a)  Across  I  versus  Within 

b)  Across  II  versus  Within 

c)  Across  III  versus  Within 

2)  Rich  Target  Prevalence 

a)  Across  I  versus  Within 

b)  Across  II  versus  Within 

c)  Across  III  versus  Within 

3)  Deficient  Target  Prevalence 

a)  Across  I  versus  Within 

b)  Across  II  versus  Within 

c)  Across  III  versus  Within 

As  each  figure  is  composed  of  four  different  bias  curves  (one  for  each  level  of 
positive  or  negative  correlation),  consider  the  following  table  for  reference  purposes. 
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Correlation  Coefficient 

Plot  Symbol 

p  =  -0.8 

x  (bold  curve) 

p  =  -0.5 

:  (double  dot) 

p  =  -0.3 

—  (double  dash) 

p  =  0 

smooth  line 

"O 

II 

O 

—  (double  dash) 

© 

II 

CL 

:  (double  dot) 

00 

© 

II 

CL 

x  (bold  curve) 

B.l  -  Enhanced  Target  Prevalence 

B.  1.1  -  Across  I  ve rsus  Within 


A=B=good  AND  p  >  0  A=B=good  AND  unequal  priors  p  >  0  A=B=fair  AND  p  >  0 


174 


0  0.1  02  0  3  0  4  05  06  07  06  00  1 


max  PQ*r(c4*  sold.  c5  =  --c6  =c7  =  x) 


A=B=fair  AND  unequal  priors  p  >  0 


0  01  02  03  04  05  06  07  06  09 


max  p<j=nc4  =  sold  c5*--.e6=  c7  =  x) 


A=good  B=fair  AND  p  >  0 


0  0.1  02  03  04  05  06  07  06  00  1 


mex  pq«r  (c4  ■  sold.  c5  ■  -  - .  c6  ■  .  c7  ■  x) 


A=good  B=poor  AND  unequal  priors  p 
>0 


A=B=poor  AND  p  >  0 


A=good  B=fair  AND  unequal  priors  p  > 
0 


A=fair  B=good  AND  p  >  0 


0  0.1  02  0  3  0  4  05  06  07  06  00  1 


max  pq=nc4  =  sold  c5  =  --,c«=c7  =  x) 


A=B=poor  unequal  priors  AND  p  >  0 


0  0.1  02  0.3  04  05  06  07  06  00  1 


max po=r  ic4  =  sold.  c5*--.e6=  c7  =  x) 


A=good  B=poor  AND  p  >  0 


0  0.1  02  03  04  0  5  06  07  0  6  00  1 


max  pq"t  (c4  -  sold.  c5  ■  •  c6  ■ : .  c7  ■  x) 


A=fair  B=good  AND  unequal  priors  p  > 
0 
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0  01  02  03  04  05  06  07  06  09 

fiwx  pQSr(c4Stoid.c5«...cfi=  c7«« 


A=fair  B=poor  AND  p  >  0 


0  0.1  02  0.3  04  05  06  07  06  00  1 

max  po=f  ic4  =  sold.  c5«...c8  =  c7«*> 


A=poor  B=good  AND  unequal  priors  p 
>0 


0  01  02  03  04  06  07  06  09 


A=B=good  AND  p  <  0 


A=fair  B=poor  AND  unequal  priors  p  > 
0 


A=poor  B=fair  AND  p  >  0 


A=B=good  AND  unequal  priors  p  <  0 


0  01  02  0.3  04  05  06  07  06  00  1 

max  pq=t  <c4  =  sold.  c5  *  •  e£  =c7  *  x) 


A=poor  B=good  AND  p  >  0 


0  01  02  0.3  04  05  06  07  06  00  1 

max  pQsr  ic4  =  sold.  eS«-.c6>:.c7«a) 


A=poor  B=fair  AND  unequal  priors  p  > 
0 


0  0.1  02  0.3  04  J.9  ^  08^  00  00  1 


A=B=fair  AND  p  <  0 
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A=B=fair  AND  unequal  priors  p  <  0 


A=B=poor  AND  p  <  0 


A=good  B=fair  AND  p  <  0 


A=good  B=fair  AND  unequal  priors  p  < 
0 


0  01  02  0.)  04  05  06  07  06  00 

maxpqerfcl  c2*  c3  =  *  c4  =  »o*d) 


A=B=poor  unequal  priors  AND  p  <  0 


0  01  02  03  04  J5  06  07  06  09 


A=good  B=poor  AND  p  <  0 


0  0.1  02  03  04  05  06  07  0  6  00  1 

mtx  pQ-r  (cl  ■  •  c2  ■  .  c3  ■  x.  c4  ■  sold) 


A=fair  B=good  AND  unequal  priors  p  < 
0 


111 


0  01  02  03  04  05  08  07  08  09 

max  pqst  (cl  *  •  -.  c2  *  ,  c3  *  ^  c4  =  sofcj) 


A=fair  B=poor  AND  p  <  0 


0  0.1  02  03  04  JS  ^  06  06  09  1 


A=poor  B=good  AND  unequal  priors  p 
<0 


0  01  02  03  04  05  06  07  06  09 

max  p**pq=r  lc4  *  sold.  cS  cB  a c7  *  x) 


A=B=good  OR  p  >  0 


A=poor  B=fair  AND  p  <  0 


A=poor  B=fair  AND  unequal  priors  p  < 
0 


A=B=good  OR  unequal  priors  p  >  0  A=B=fair  OR  p  >  0 
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A=B=fair  OR  unequal  priors  p  >  0  A=B=poor  OR  p  >  0 


A=good  B=fair  OR  p  >  0 


A=good  B=fair  OR  unequal  priors  p  >  0 


A=B=poor  unequal  priors  OR  p  >  0 


0  0.1  02  0.3  04  05  06  07  06  00  1 

nwx  p»(*po=r  ic4  =  sold.  c5  =  •  -.  c6  =  c7  =  x) 


A=good  B=poor  OR  p  >  0 


0  0.1  0  2  03  04  03  06  07  0  6  0.6  1 

mtx  p+<Kxrr  <c4  ■  sold.  c5  ■  •  c6  ■  .  c7  ■  x) 


A=fair  B=good  OR  unequal  priors  p  >  0 
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A=fair  B=poor  OR  p  >  0  A=fair  B=poor  OR  unequal  priors  p  >  0 


A=poor  B=good  OR  p  >  0 


A  poor  B  good  OR^  unequal  priors  p  >  A=poor  B=fair  OR  p  >  0  A=poor  B=fair  OR  unequal  priors  p  >  0 


A=B=good  OR  p  <  0  A=B=good  OR  unequal  priors  p  <  0  A=B=fair  OR  p  <  0 
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A=B=fair  OR  unequal  priors  p  <  0 


A=B=poor  OR  p  <  0 


A=B=poor  unequal  priors  OR  p  <  0 


A=good  B=fair  OR  p  <  0  A=good  B=fair  OR  unequal  priors  p  <  0 


A=good  B=poor  OR  p  <  0 


A=good  B=poor  OR  unequal  priors  p  < 
0 


A=fair  B=good  OR  p  <  0  A=fair  B=good  OR  unequal  priors  p  <  0 
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A=fair  B=poor  OR  p  <  0  A=fair  B=poor  OR  unequal  priors  p  <  0 


B.1.2  -  Across  II  versus  Within 


A=B=good  AND  p  >  0  A=B=good  AND  unequal  priors  p  >  0 


0  01  02  03  04  05  08  07  08  09  1 


A=poor  B=good  OR  p  <  0 


0  0.1  02  0.3  04  05  ^0.8  07^  00  09 


A=poor  B=fair  OR  unequal  priors  p  <  0 


Bia*  between  A2  W  A=B=f»ir  po»  correlation* 


A=B=fair  AND  p  >  0 
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A=B=fair  AND  unequal  priors  p  >  0 


A=B=poor  AND  p  >  0 


A=B=poor  unequal  priors  AND  p  >  0 


A=good  B=fair  AND  p  >  0 


A=good  B=fair  AND  unequal  priors  p  > 
0 


A=good  B=poor  AND  p  >  0 


A=good  B=poor  AND  unequal  priors  p 
>0 


A=fair  B=good  AND  p  >  0 


A=fair  B=good  AND  unequal  priors  p  > 
0 
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A=fair  B=poor  AND  p  >  0 


A=fair  B=poor  AND  unequal  priors  p  > 
0 


A=poor  B=good  AND  p  >  0 


A=B=good  AND  p  <  0 


A=B=good  AND  unequal  priors  p  <  0 


A=B=fair  AND  p  <  0 
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A=B=fair  AND  unequal  priors  p  <  0 


A=B=poor  AND  p  <  0 


A=B=poor  unequal  priors  AND  p  <  0 


A=good  B=poor  AND  unequal  priors  p 
<0 


A=fair  B=good  AND  p  <  0 


A=fair  B=good  AND  unequal  priors  p  < 
0 
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A=fair  B=poor  AND  p  <  0 


A=poor  B=good  AND  unequal  priors  p 
<0 


A=B=good  OR  p  >  0 


A=poor  B=fair  AND  p  <  0 


A=poor  B=fair  AND  unequal  priors  p  < 
0 


A=B=good  OR  unequal  priors  p  >  0 


A=B=fair  OR  p  >  0 
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A=B=fair  OR  unequal  priors  p  >  0  A=B=poor  OR  p  >  0 


A=good  B=fair  OR  p  >  0  A=good  B=fair  OR  unequal  priors  p  >  0 


A=B=poor  unequal  priors  OR  p  >  0 


0  0.1  02  0.3  04  05  06  07  06  00  1 

mn  p*<w>Q=r  ic4  =  sold.  cS  « •  c6  » :.  c7  «  « 


A=good  B=poor  OR  p  >  0 


A=fair  B=good  OR  unequal  priors  p  >  0 
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A=fair  B=poor  OR  p  >  0  A=fair  B=poor  OR  unequal  priors  p  >  0 


A=poor  B=good  OR  unequal  priors  p  > 
0 


A=poor  B=fair  OR  p  >  0 


A=B=good  OR  p  <  0 


A=B=good  OR  unequal  priors  p  <  0 


A=poor  B=good  OR  p  >  0 


A=poor  B=fair  OR  unequal  priors  p  >  0 


A=B=fair  OR  p  <  0 
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A=good  B=fair  OR  p  <  0  A=good  B=fair  OR  unequal  priors  p  <  0 


0  01  02  0.3  04  05  06  07  06  09 


A=B=poor  unequal  priors  OR  p  <  0 


0  01  02  03  04  05  06  07  06  09 


A=good  B=poor  OR  p  <  0 


ma*  p»<mkjt  (cl  -  - c2  -  ,  c3  •  k.  c«  -  soW) 


A=fair  B=good  OR  unequal  priors  p  <  0 
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A=fair  B=poor  OR  p  <  0  A=fair  B=poor  OR  unequal  priors  p  <  0 


A=poor  B=good  OR  unequal  priors  p  < 
0 


A=poor  B=fair  OR  p  <  0 


B.1.3  -  Across  III  versus  Within 


A=B=good  AND  p  >  0 


A=B=good  AND  unequal  priors  p  >  0 


0  01  02  03  04  05  06  07  06  09 


A=poor  B=good  OR  p  <  0 


0  01  0  2  0  3  0  4  0  5  0  6  0  7  06  0  9  1 


A=poor  B=fair  OR  unequal  priors  p  <  0 


Bia*  betwen  A3  v»  W  A=B-fa»  po$  correitliom 


A=B=fair  AND  p  >  0 
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A=good  B=poor  AND  unequal 
priors  p  >  0 


A=fair  B=good  AND  p  >  0 


A=fair  B=good  AND  unequal 
priors  p  >  0 
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A=fair  B=poor  AND  p  >  0  A=fair  B=poor  AND  unequal  priors  p  >  0  A=poor  B=good  AND  p  >  0 


A=poor  B=good  AND  unequal 
priors  p  >  0 


A=poor  B=fair  AND  p  >  0 


A=poor  B=fair  AND  unequal 
priors  p  >  0 


A=B=good  AND  p  <  0 


A=B=good  AND  unequal  priors  p 
<  0 


A=B=fair  AND  p  <  0 
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02 


02  0.3 


A=B=fair  AND  unequal  priors  p 
<0 


0  01  02  03  04  05  0^6  07  00  09 


A=good  B=fair  AND  p  <  0 


0  0.1  0  2  03  04  05  06  07  0  6  00 

mtx  pq*r  (cl  ■  c2  ■ c3  •  x) 


A=good  B=poor  AND  unequal 
priors  p  <  0 


A=B=poor  AND  p  <  0 


A=B=poor  unequal  priors  AND  p 
<0 


A=good  B=fair  AND  unequal 
priors  p  <  0 


A=good  B=poor  AND  p  <  0 


A=fair  B=good  AND  p  <  0 


A=fair  B=good  AND  unequal 
priors  p  <  0 
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A=fair  B=poor  AND  p  <  0  ^  unequal  A=poor  B=good  AND  p  <  0 


A=poor  B=good  AND  unequal 
priors  p  <  0 


A=poor  B=fair  AND  p  <  0 


A=poor  B=fair  AND  unequal 
priors  p  <  0 


A=B=good  OR  p  >  0  A=B=good  OR  unequal  priors  p  >  0  A=B=fair  OR  p  >  0 
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A=B=fair  OR  unequal  priors  p  >  0  A=B=poor  OR  p  >  0 


A=good  B=fair  OR  p  >  0  A=good  B=fair  OR  unequal  priors  p  >  0 


A=B=poor  unequal  priors  OR  p  >  0 


0  01  02  03  04  05  06  07  06  09 

rrmx  p*q-p<*='  <cS  *  OB  * e7  *  x> 


A=good  B=poor  OR  p  >  0 


B<»s  bctwtcn  A3  vt  W  A=f«r  B-good  po»  corr^ahont  irequai  poor*  OR 


0  0.1  02  03  04  05  06  07  06  00  1 

max  p*roq-r  <c5  ■  c6  •  .  c7  ■  x) 


A=fair  B=good  OR  unequal  priors  p  >  0 
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A=fair  B=poor  OR  p  >  0 


A=fair  B=poor  OR  unequal  priors  p  >  0 


0  01  02  0.3  0  4  05  06  07  06  00  1 

rmx  p*q-po=r  <c5  *  06  « c7  ■  x> 


A=poor  B=good  OR  p  >  0 


0  01  02  03  04  05  06  07  06  09 

rmx  <c5  *  c6  ■ e7  *  x> 


A=poor  B=fair  OR  unequal  priors  p  >  0 


A=B=fair  OR  p  <  0 
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A=B=fair  OR  unequal  priors  p  <  0  A=B=poor  OR  p  <  0 


A=good  B=fair  OR  p  <  0  A=good  B=fair  OR  unequal  priors  p  <  0 


0  01  02  OS  04  05  06  07  08  09 

max  C2  «  cS  « x> 


A=B=poor  unequal  priors  OR  p  <  0 


0  0*1  02  0*3  04  0  5  0  6  0  7  08  0  9  1 

rmx  p*«H>q=r  <c1  *  c2  ■  .  cS  »  x) 


A=good  B=poor  OR  p  <  0 


0  0.1  02  OS  04  05  06  07  06  09 

max  p*<Hxrr  (cl  ■  c2  ■  .  c3  •  X) 


A=fair  B=good  OR  unequal  priors  p  <  0 
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A=fair  B=poor  OR  p  <  0 


A=fair  B=poor  OR  unequal  priors  p  <  0 


A=poor  B=good  OR  p  <  0 


A=poor  B=good  OR  unequal  priors  p  < 
0 


A=poor  B=fair  OR  p  <  0 


A=poor  B=fair  OR  unequal  priors  p  <  0 


B.2  —  Rich  Target  Prevalence 

B.2.1  -  Across  I  versus  Within 


A=B=good  AND  p  >  0 


A=B=good  AND  unequal  priors  p  >  0 


A=B=fair  AND  p  >  0 
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A=B=fair  AND  unequal  priors  p  >  0 


A=B=poor  AND  p  >  0 


0  01  02  03  04  05  06  07  06  09 

m«  PQSMC4*  soM.  c5  c7«*> 


A=B=poor  unequal  priors  AND  p  >  0 


0  0.1  02  03  04  05  06  07  06  09  1 

mu  po=f  ic4  =  sold.  c5«...cfi=  c7«*) 


A=good  B=poor  AND  p  >  0 


0  0.1  02  03  04  05  06  07  06  09  1 

max  p<rr  (c4  ■  sold.  c5  ■  c6  c7  ■  x) 


A=fair  B=good  AND  unequal  priors  p  > 
0 
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0  0.1  02  03  04  OS  06  07  06  00  1 

m«pQ=nc4  =  so4<lc5«..1c6=c7*i) 


A=fair  B=poor  AND  p  >  0 


0  0.1  02  0.3  04  OS  06  07  06  00  1 

max  po=f  ic4  =  sold.  c5«.-.c6=  c7«« 


A=poor  B=good  AND  unequal  priors  p 
>0 


0  01  02  03  04  06  07  06  09 


A=B=good  AND  p  <  0 


A=fair  B=poor  AND  unequal  priors  p  > 
0 


A=poor  B=fair  AND  p  >  0 


A=B=good  AND  unequal  priors  p  <  0 


0  0.1  02  0.3  04  OS  06  07  06  00  1 

max  pq=t  <c4  =  sold.  c5  *  •  e£  =c7  *  U 


A=poor  B=good  AND  p  >  0 


0  0.1  02  0.3  04  OS  06  07  06  00  1 

max  po=f  ic4  =  sold.  cS«.-.c6=  c7«x) 


A=poor  B=fair  AND  unequal  priors  p  > 
0 


•  0.1  02  0.3  04  J.9  ^  08^  06  Of  1 


A=B=fair  AND  p  <  0 
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A=B=fair  AND  unequal  priors  p  <  0 


A=B=poor  AND  p  <  0 


A=B=poor  unequal  priors  AND  p  <  0 


A=good  B=fair  AND  p  <  0 


A=good  B=fair  AND  unequal  priors  p  < 
0 


A=good  B=poor  AND  p  <  0 
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0  01  02  03  04  05  08  07  08  09 

max  pqst  (cl  *  •  -.  c2  *  ,  c3  *  ^  c4  =  sofcj) 


A=fair  B=poor  AND  p  <  0 


0  0.1  02  03  04  ^  06  08  09  1 


A=poor  B=good  AND  unequal  priors  p 
<0 


A=B=good  OR  p  >  0 


A=B=good  OR  unequal  priors  p  >  0 


A=B=fair  OR  p  >  0 
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A=B=fair  OR  unequal  priors  p  >  0  A=B=poor  OR  p  >  0 


A=good  B=fair  OR  p  >  0 


A=good  B=fair  OR  unequal  priors  p  >  0 


A=B=poor  unequal  priors  OR  p  >  0 


0  0.1  02  0.3  04  03  06  07  06  00  1 

mn  p*<w>Q=r  ic4  =  sold.  cS  « •  c6  » :.  c7  «  x) 


A=good  B=poor  OR  p  >  0 


0  0.1  0  2  03  04  03  06  0  7  0  6  00  1 

mtx  p+<Kxrr  <c4  ■  sold.  cS  •  •  *.  c6  » c7  ■  x) 


A=fair  B=good  OR  unequal  priors  p  >  0 
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0  0.1  02  0.3  0  4  05  06  07  06  00 

mu  p*o«x*=r  lc4  =  sold.  cS  « •  •.  c6  *  c7  *  x) 


0  0.1  02  03  04  05  06  07  06  00 

mu  p*<hh*=Mc4  =  »oM.  cS  « •  c«  s  .  c7  *  x) 


A=fair  B=poor  OR  p  >  0  A=fair  B=poor  OR  unequal  priors  p  >  0 


A=poor  B=good  OR  unequal  priors  p  > 
0 


A=poor  B=fair  OR  p  >  0 


A=B=good  OR  p  <  0  A=B=good  OR  unequal  priors  p  <  0 


0  0.1  02  03  04  05  06  07  06  00 

mu  p*<HH*=r  ic4  =  »oM.  cS  « •  c6  s  .  c7  *  x) 


A=poor  B=good  OR  p  >  0 


0  0.1  02  0.3  04  05  06  07  06  00  1 

mu  p^HK»=r  IC4  =  sold.  c5  « •  c6  » c7  *  x) 


A=poor  B=fair  OR  unequal  priors  p  >  0 


Bas  between  A1  v»  W  A=B=f«r  neo  coneMon  ntfi  OR 


A=B=fair  OR  p  <  0 
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A=B=fair  OR  unequal  priors  p  <  0 


A=B=poor  OR  p  <  0 


A=B=poor  unequal  priors  OR  p  <  0 


A=good  B=fair  OR  p  <  0  A=good  B=fair  OR  unequal  priors  p  <  0 


A=good  B=poor  OR  p  <  0 


A=good  B=poor  OR  unequal  priors  p  < 
0 


A=fair  B=good  OR  p  <  0  A=fair  B=good  OR  unequal  priors  p  <  0 


205 


A=fair  B=poor  OR  p  <  0 


A=fair  B=poor  OR  unequal  priors  p  <  0 


A=poor  B=good  OR  p  <  0 


A=poor  B=good  OR  unequal  priors  p  < 
0 


A=poor  B=fair  OR  p  <  0  A=poor  B=fair  OR  unequal  priors  p  <  0 


B.2.2  -  Across  II  versus  Within 


A=B=good  AND  p  >  0 


A=B=good  AND  unequal  priors  p  >  0 


A=B=fair  AND  p  >  0 
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A=B=fair  AND  unequal  priors  p  >  0 


A=B=poor  AND  p  >  0 


A=B=poor  unequal  priors  AND  p  >  0 


A=good  B=fair  AND  p  >  0 


A=good  B=fair  AND  unequal  priors  p  > 
0 


A=good  B=poor  AND  p  >  0 


A=good  B=poor  AND  unequal  priors  p 
>0 


A=fair  B=good  AND  p  >  0 


A=fair  B=good  AND  unequal  priors  p  > 
0 
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0  01  02  03  04  OS  06  07  06  0  9  1 

m«x  pq=t  (c4  =  sold.  c5  *  •  -,  cfi  =  c7  *  x) 


A=fair  B=poor  AND  p  >  0 


A=poor  B=good  AND  unequal  priors  p 
>  0 


A=B=good  AND  p  <  0 


A=fair  B=poor  AND  unequal  priors  p  > 
0 


A=poor  B=fair  AND  p  >  0 


A=B=good  AND  unequal  priors  p  <  0 


A=poor  B=good  AND  p  >  0 


mu  p<j=r  ic4  S  sold.  c5  s- c6  =  c7  =  x) 


A=poor  B=fair  AND  unequal  priors  p  > 
0 


A=B=fair  AND  p  <  0 
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A=B=fair  AND  unequal  priors  p  <  0 


A=B=poor  AND  p  <  0 


A=B=poor  unequal  priors  AND  p  <  0 


A=good  B=fair  AND  p  <  0 


A=good  B=fair  AND  unequal  priors  p  < 
0 


A=good  B=poor  AND  p  <  0 


A=good  B=poor  AND  unequal  priors  p 
<0 


A=fair  B=good  AND  p  <  0 


A=fair  B=good  AND  unequal  priors  p  < 
0 
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A=poor  B=good  AND  unequal  priors  p 
<0 


A=poor  B=fair  AND  p  <  0 


A=B=good  OR  p  >  0 


A=B=good  OR  unequal  priors  p  >  0 


A=poor  B=good  AND  p  <  0 


A=poor  B=fair  AND  unequal  priors  p  < 
0 


Bas  between  A2  v»  W  A=B=f air  po»  cooetabon  rich  OR 


A=B=fair  OR  p  >  0 
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i  W  A= B=fa*  pot  c 


A=B=fair  OR  unequal  priors  p  >  0 


A=B=poor  OR  p  >  0 


A=B=poor  unequal  priors  OR  p  >  0 


A=good  B=fair  OR  p  >  0  A=good  B=fair  OR  unequal  priors  p  >  0 


A=good  B=poor  OR  p  >  0 


A=good  B=poor  OR  unequal  priors  p  > 
0 


A=fair  B=good  OR  p  >  0  A=fair  B=good  OR  unequal  priors  p  >  0 
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A=fair  B=poor  OR  p  >  0  A=fair  B=poor  OR  unequal  priors  p  >  0  A=poor  B=good  OR  p  >  0 


A=poor  B=good  OR  unequal  priors  p  > 
0 


A=poor  B=fair  OR  p  >  0  A=poor  B=fair  OR  unequal  priors  p  >  0 


A=B=good  OR  p  <  0 


A=B=good  OR  unequal  priors  p  <  0 


A=B=fair  OR  p  <  0 
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A=B=fair  OR  unequal  priors  p  <  0 


A=B=poor  OR  p  <  0 


A=B=poor  unequal  priors  OR  p  <  0 
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A=fair  B=poor  OR  p  <  0  A=fair  B=poor  OR  unequal  priors  p  <  0 


A=poor  B=good  OR  unequal  priors  p  < 
0 


A=poor  B=fair  OR  p  <  0 


B.2.3  -  Across  III  versus  Within 


A=B=good  AND  p  >  0  A=B=good  AND  unequal  priors  p  >  0 


A=poor  B=good  OR  p  <  0 


0  01  02  03  0  4  05  0  6  0  7  0  6  0  9 


A=poor  B=fair  OR  unequal  priors  p  <  0 


Bias  between  A3  vs  W  A=B=fau  po»  c 


A=B=fair  AND  p  >  0 
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A=good  B=fair  AND  p  >  0 


A=good  B=fair  AND  unequal  priors  p  >  0 


A=good  B=poor  AND  unequal 
priors  p  >  0 


A=fair  B=good  AND  p  >  0 


0  01  02  03  04  05  06  07  06  09 

max  pqst  ic4  =  sold.  cSa--,c8«:.c7«D 


A=B=poor  unequal  priors  AND  p 
>0 


0  01  02  03  04  05  06  07  06  09 

rraxpq=r  (c4  =  sold,  c5«*-.  e6=  c7*x) 


A=good  B=poor  AND  p  >  0 


0  01  02  03  04  05  0*6  07  06  09 

mex  pq*r  (c4  ■  sold.  c5  ■  • c6  ■  .  c7  ■  x) 


A=fair  B=good  AND  unequal 
priors  p  >  0 
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A=fair  B=poor  AND  p  >  0 


A=fair  B=poor  AND  unequal  priors  p  >  0 


A=poor  B=good  AND  p  >  0 


A=B=good  AND  p  <  0 


A=B=good  AND  unequal  priors  p 
<  0 


A=B=fair  AND  p  <  0 
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0  01  02  0)  04  OS  06  07  06  00 

max  pqst  (cl  *  •  -.  c2  *  ,  c3  *  ^  c4  =  sofcj) 


A=B=fair  AND  unequal  priors  p 
<0 


0  01  02  03  04  J5  06  07  06  09 


A=good  B=fair  AND  p  <  0 


0  0.1  0  2  03  04  OS  06  0.7  0  6  09 

max  pq*r  (cl  ■  -  c2  ■  .  c3  •  x.  c4  ■  sold) 


A=good  B=poor  AND  unequal 
priors  p  <  0 


A=B=poor  AND  p  <  0 


A=B=poor  unequal  priors  AND  p 
<0 


A=good  B=fair  AND  unequal  .  ,  „  . n 

&  A=good  B=poor  AND  p  <  0 

priors  p  <  0 


A=fair  B=good  AND  p  <  0 


A=fair  B=good  AND  unequal 
priors  p  <  0 
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A=fair  B=poor  AND  p  <  0 


A=fair  B=poor  AND  unequal 
priors  p  <  0 


A=poor  B=good  AND  p  <  0 


A=poor  B=good  AND  unequal 
priors  p  <  0 


A=poor  B=fair  AND  p  <  0 


A=poor  B=fair  AND  unequal 
priors  p  <  0 


A=B=good  OR  p  >  0  A=B=good  OR  unequal  priors  p  >  0  A=B=fair  OR  p  >  0 
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A=B=fair  OR  unequal  priors  p  >  0  A=B=poor  OR  p  >  0 


A=good  B=fair  OR  p  >  0  A=good  B=fair  OR  unequal  priors  p  >  0 


A=good  B=poor  OR  unequal  priors  p  > 
0 


A=fair  B=good  OR  p  >  0 


A=B=poor  unequal  priors  OR  p  >  0 


0  0.1  02  0.3  04  05  06  07  06  00  1 

max  p»(*po=r  ic4  =  sold.  c5  =  •  -.  c6  =  c7  =  x) 


A=good  B=poor  OR  p  >  0 


0  0.1  0  2  03  04  03  06  07  0  6  00  1 

max  p+<Kxrr  <c4  ■  sold.  c5  ■  •  c6  ■  .  c7  ■  x) 


A=fair  B=good  OR  unequal  priors  p  >  0 
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0  01  02  03  04  05  06  07  06  09 

mu  p*o«x*=r  lc4  =  sold.  cS  « •  •.  c6  *  c7  *  x) 


A=fair  B=poor  OR  p  >  0 


0  01  02  03  04  05  06  07  06  09 

iw  P+WF*  ic4  =  sold.  cS  « •  c6  » c7  *  x) 


A=poor  B=good  OR  unequal  priors  p  > 
0 


Bit*  between  A3  v*  W  A=B=good  neg  correlation  heft  OR 


A=B=good  OR  p  <  0 


A=fair  B=poor  OR  unequal  priors  p  >  0 


A=poor  B=fair  OR  p  >  0 


A=B=good  OR  unequal  priors  p  <  0 


0  01  02  0  3  0  4  05  06  07  06  00  1 

mu  p*<HH*=r  ic4  =  »oM.  cS  « •  c6  s  .  c7  *  *) 


A=poor  B=good  OR  p  >  0 


0  01  02  0.3  04  05  06  07  06  00  1 

mu  p^HK»=r  IC4  =  sold.  c5  « •  c6  » c7  *  x) 


A=poor  B=fair  OR  unequal  priors  p  >  0 


B»s  between  A3  vs  W  A=B=f air  nec  correlation  nc*i  OR 


A=B=fair  OR  p  <  0 
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0  0.1  02  OS  04  M  JM  07  ^  04  04  1 


A=B=fair  OR  unequal  priors  p  <  0 


0  0*1  '  o':  0  3  0  4  05  06  0  7  06  09  I 


A=good  B=fair  OR  p  <  0 


A=good  B=poor  OR  unequal  priors  p  < 
0 


A=B=poor  OR  p  <  0 


A=B=poor  unequal  priors  OR  p  <  0 


A=good  B=fair  OR  unequal  priors  p  <  0 


A=good  B=poor  OR  p  <  0 


A=fair  B=good  OR  p  <  0  A=fair  B=good  OR  unequal  priors  p  <  0 
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A=fair  B=poor  OR  p  <  0 


A=fair  B=poor  OR  unequal  priors  p  <  0 


A=poor  B=good  OR  unequal  priors  p  < 
0 


A=poor  B=fair  OR  p  <  0 


B.3  —  Deficient  Target  Prevalence 

B.  3. 1  -  Across  I  versus  Within 


A=B=good  AND  p  >  0 


A=B=good  AND  unequal  priors  p  >  0 


0  01  02  03  04  05  06  07  06  09 


A=poor  B=good  OR  p  <  0 


©  0.1  0.2  0.3  04  M  J>8  ^  07^  00  00 


A=poor  B=fair  OR  unequal  priors  p  <  0 


0  01  or  03  04  05  06  07  06  09  1 

mupo=r  <c4  =  soid,  c5«-..c6=  c7**) 


A=B=fair  AND  p  >  0 
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A=B=fair  AND  unequal  priors  p  >  0  A=B=poor  AND  p  >  0  A=B=poor  unequal  priors  AND  p  >  0 


A=good  B=fair  AND  p  >  0  A=g0°d  B=fair^  unc<*ual  Priors  <*  ^  A=good  B=poor  AND  p  >  0 


A=good  B=poor  AND  unequal  priors  p 
>0 


A=fair  B=good  AND  p  >  0 


A=fair  B=good  AND  unequal  priors  p  > 
0 
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0  2 


A=fair  B=poor  AND  p  >  0 


A=poor  B=good  AND  unequal  priors  p 
>0 


0  01  02  03  04  06  07  06  09 


A=B=good  AND  p  <  0 


A=B=good  AND  unequal  priors  p  <  0 


A=B=fair  AND  p  <  0 
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A=B=fair  AND  unequal  priors  p  <  0 


A=B=poor  AND  p  <  0 


A=B=poor  unequal  priors  AND  p  <  0 


A=good  B=fair  AND  p  <  0  A=g0°d  B=fair^  unc<*ual  Priors  P  5  A=good  B=poor  AND  p  <  0 


A=good  B=poor  AND  unequal  priors  p 
<0 


A=fair  B=good  AND  p  <  0 


A=fair  B=good  AND  unequal  priors  p  < 
0 
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0  01  02  03  04  05  08  07  08  09 

max  pqst  (cl  *  •  -.  c2  *  ,  c3  *  x  c4  =  so*d) 


A=fair  B=poor  AND  p  <  0 


0  01  02  03  04  J5  06  07  08  09 


A=poor  B=good  AND  unequal  priors  p 
<  0 


0  01  02  03  04  05  06  07  08  09 

max  p**pq=r  lc4  *  sold.  cS  c6  a e7  *  x) 


A=B=good  OR  p  >  0 


A=poor  B=fair  AND  p  <  0 


A=poor  B=fair  AND  unequal  priors  p  < 
0 


A=B=good  OR  unequal  priors  p  >  0  A=B=fair  OR  p  >  0 
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A=B=fair  OR  unequal  priors  p  >  0  A=B=poor  OR  p  >  0 


A=good  B=fair  OR  p  >  0  A=good  B=fair  OR  unequal  priors  p  >  0 


A=B=poor  unequal  priors  OR  p  >  0 


0  01  02  03  04  05  06  07  06  09 

max  p»(*po=r  ic4  =  sold.  c5  =  •  -.  c6  =  c7  =  x) 


A=good  B=poor  OR  p  >  0 


0  0.1  0  2  03  04  0  3  0  6  0  7  0  6  09 

max  p*<Hxrr  <c4  ■  sold.  c5  ■  •  c6  ■  .  c7  ■  x) 


A=fair  B=good  OR  unequal  priors  p  >  0 
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A=fair  B=poor  OR  p  >  0  A=fair  B=poor  OR  unequal  priors  p  >  0 


A=poor  B=good  OR  p  >  0 


A=poor  B=good  OR  unequal  priors  p  > 
0 


A=poor  B=fair  OR  p  >  0  A=poor  B=fair  OR  unequal  priors  p  >  0 


A=B=good  OR  p  <  0  A=B=good  OR  unequal  priors  p  <  0 


A=B=fair  OR  p  <  0 
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A=B=fair  OR  unequal  priors  p  <  0  A=B=poor  OR  p  <  0 


A=good  B=fair  OR  p  <  0  A=good  B=fair  OR  unequal  priors  p  <  0 


0  01  02  03  04  05  00  07  00  00 


A=B=poor  unequal  priors  OR  p  <  0 


0  0.1  02  03  04  05  08  07  08  09 


A=good  B=poor  OR  p  <  0 


0  0.1  0.2  03  04  05  06  07  06  09 

m«xp*<H><rr<c1  ■  -  *.  c2  ■  c3  ■  x.  o4  ■  soM) 


A=fair  B=good  OR  unequal  priors  p  <  0 
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A=fair  B=poor  OR  p  <  0  A=fair  B=poor  OR  unequal  priors  p  <  0 


A=poor  B=good  OR  p  <  0 


A=poor  B=good  OR  unequal  priors  p  < 
0 


A=poor  B=fair  OR  p  <  0 


A=poor  B=fair  OR  unequal  priors  p  <  0 


B.3.2  -  Across  II  versus  Within 


A=B=good  AND  p  >  0 


A=B=good  AND  unequal  priors  p  >  0 


A=B=fair  AND  p  >  0 


230 


A=B=fair  AND  unequal  priors  p  >  0 


A=B=poor  AND  p  >  0 


A=B=poor  unequal  priors  AND  p  >  0 


A=good  B=fair  AND  p  >  0 


A=good  B=poor  AND  unequal  priors  p 
>0 


A=good  B=fair  AND  unequal  priors  p  > 
0 


max  pq»r  (c<  -  sola.  cS  ■  -  c6  ■  .  c7  ■  x) 


A=fair  B=good  AND  p  >  0 


A=good  B=poor  AND  p  >  0 


. . . o,? 

msxpq-rics  ■  sold.  cS  ■  -  c«  ■  .  c7  •  x) 


A=fair  B=good  AND  unequal  priors  p  > 
0 
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A=poor  B=good  AND  unequal  priors  p 
>0 


A=poor  B=fair  AND  p  >  0 


A=B=good  AND  p  <  0 


A=B=good  AND  unequal  priors  p  <  0 


A=poor  B=good  AND  p  >  0 


mu  pQ=r  lc4  s  sold.  c5  =  -  c6  =  c7  =  x) 


A=poor  B=fair  AND  unequal  priors  p  > 
0 


A=B=fair  AND  p  <  0 
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A=B=fair  AND  unequal  priors  p  <  0 


A=B=poor  AND  p  <  0 


A=B=poor  unequal  priors  AND  p  <  0 
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A=poor  B=good  AND  unequal  priors  p 
<0 


A=poor  B=fair  AND  p  <  0 


A=poor  B=fair  AND  unequal  priors  p  < 
0 


i  def  unequtl  poors  Oft 


(All  values  >  0)  A=B=good  OR  p  >  0 


(All  values  >  0)  A=B=good  OR  unequal 
priors  p  >  0 


A=B=fair  OR  p  >  0 
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tA2v*  WA=B=poorpo*c 


A=B=fair  OR  unequal  priors  p  >  0 


0  01  02  03  04  05  0*6 0*7  08  09 

max  p»o-pQ=r  ic4  =  sold.  c5  *  •  •.  c6  s  c7  *  x) 


A=good  B=fair  OR  p  >  0 


A=good  B=poor  OR  unequal  priors  p  > 
0 


A=B=poor  OR  p  >  0 


A=good  B=fair  OR  unequal  priors  p  >  0 


Bui  between  A2  vs  W  A»f  »ir 


0  01  02  03  04  05  Ofi  Of  Oft  09  1 

max  p«o-cQf  <c4  ■  sold.  c$  •  •  *.  c6  ■ c7  ■  x) 


A=fair  B=good  OR  p  >  0 


A=B=poor  unequal  priors  OR  p  >  0 


0  0.1  02  03  04  05  06  07  06  09  1 

mat  p*<*pq=r  ic4  =  sold.  c5«--.  c6=  c7  =  x) 


A=good  B=poor  OR  p  >  0 


mtx  p»<nxrf  <e«  •  sow.  cS  ■  •  c6  • c7  ■  » 


A=fair  B=good  OR  unequal  priors  p  >  0 
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025 


A=poor  B=good  OR  unequal  priors  p  > 
0 


A=poor  B=fair  OR  p  >  0  A=poor  B=fair  OR  unequal  priors  p  >  0 


A=B=good  OR  p  <  0 


A=B=good  OR  unequal  priors  p  <  0 


A=B=fair  OR  p  <  0 
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A=good  B=fair  OR  p  <  0  A=good  B=fair  OR  unequal  priors  p  <  0 


A=good  B=poor  OR  p  <  0 
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0.25 


A=fair  B=poor  OR  p  <  0 


A=fair  B=poor  OR  unequal  priors  p  <  0 


A=poor  B=good  OR  p  <  0 


A=poor  B=fair  OR  unequal  priors  p  <  0 


A=poor  B=good  OR  unequal  priors  p  < 
0 


A=poor  B=fair  OR  p  <  0 


B.3.3  -  Across  III  versus  Within 


A=B=good  AND  p  >  0 


A=B=good  AND  unequal  priors  p  >  0 


A=B=fair  AND  p  >  0 


238 


A=good  B=fair  AND  p  >  0 


A=good  B=fair  AND  unequal  priors  p  >  0  A=good  B=poor  AND  p  >  0 


A=good  B=poor  AND  unequal 
priors  p  >  0 


A=fair  B=good  AND  p  >  0 


A=fair  B=good  AND  unequal 
priors  p  >  0 
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A=fair  B=poor  AND  p  >  0 


A=fair  B=poor  AND  unequal  priors  p  >  0 


A=poor  B=good  AND  p  >  0 


A=poor  B=good  AND  unequal 
priors  p  >  0 


A=poor  B=fair  AND  p  >  0 


A=poor  B=fair  AND  unequal 
priors  p  >  0 


A=B=good  AND  p  <  0 


A=B=good  AND  unequal  priors  p 
<  0 


A=B=fair  AND  p  <  0 
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A=good  B=fair  AND  p  <  0  ^  good  B  fair  AND  unequal  A=good  B=poor  AND  p  <  0 

F  “  priors  p  <  0  K  “ 


A=good  B=poor  AND  unequal 
priors  p  <  0 


A=fair  B=good  AND  p  <  0 


A=fair  B=good  AND  unequal 
priors  p  <  0 
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A=fair  B=poor  AND  p  <  0 


A=fair  B=poor  AND  unequal 
priors  p  <  0 


A=poor  B=good  AND  p  <  0 


A=poor  B=good  AND  unequal 
priors  p  <  0 


A=poor  B=fair  AND  p  <  0 


A=poor  B=fair  AND  unequal 
priors  p  <  0 


A=B=good  OR  p  >  0 


A=B=good  OR  unequal  priors  p  >  0 


A=B=fair  OR  p  >  0 
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A=good  B=fair  OR  p  >  0  A=good  B=fair  OR  unequal  priors  p  >  0  A=good  B=poor  OR  p  >  0 


A  good  B  poor  OR^  unequal  priors  p  >  A=fair  B=good  OR  p  >  0  A=fair  B=good  OR  unequal  priors  p  >  0 
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0  0.1  02  OS  0  4  05  06  07  06  00 

mu  p*o«x*=r  lc4  =  sold.  cS  « •  •.  c6  *  c7  *  x) 


A=fair  B=poor  OR  p  >  0 


0  01  02  OS  04  05  06  07  06  00  1 

iw  P+WF*  ic4  =  sold,  cS  « •  e6  » c7  «  x) 


A=poor  B=good  OR  unequal  priors  p  > 
0 


i  A3  vs  W  A=B=oood  neo  corrHalion  def  OR 


A=B=good  OR  p  <  0 


A=fair  B=poor  OR  unequal  priors  p  >  0 


A=poor  B=fair  OR  p  >  0 


A=B=good  OR  unequal  priors  p  <  0 


0  0.1  02  OS  04  05  06  07  06  00  1 


A=poor  B=good  OR  p  >  0 


0  0.1  02  OS  04  05  06  07  06  00  1 


mu  p*i«x*=nc4  =  sold,  cS  =  ■ -.  c6  »  c7  =  x) 


A=poor  B=fair  OR  unequal  priors  p  >  0 


Bios  between  A3  vs  W  A=B=feir  neo  corretation  def  OR 


A=B=fair  OR  p  <  0 
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A=B=fair  OR  unequal  priors  p  <  0 


A=B=poor  OR  p  <  0 


A=B=poor  unequal  priors  OR  p  <  0 


A=good  B=fair  OR  p  <  0  A=good  B=fair  OR  unequal  priors  p  <  0 


A=good  B=poor  OR  p  <  0 


A=good  B=poor  OR  unequal  priors  p  < 
0 


A=fair  B=good  OR  p  <  0  A=fair  B=good  OR  unequal  priors  p  <  0 
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A=fair  B=poor  OR  p  <  0 


A=fair  B=poor  OR  unequal  priors  p  <  0  A=poor  B=good  OR  p  <  0 
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Appendix  C  Code 


Included  in  this  appendix  is  all  of  the  code  used  to  run  the  simulation.  First,  a 
brief  discussion  of  the  algorithms  used  to  create  the  simulation  data  is  given.  Following 
this  section  is  the  actual  source  code  itself. 

C.  1  Outline  of  Program 

C.1.1  Across  I  simulation 

The  following  contains  a  brief  outline  of  the  algorithm  that  has  been  developed  to 
generate  the  ROC  curves  of  interest  for  across  I  combined  classification  systems.  In  the 
text  below,  those  items  written  in  courier  new  refer  to  either  variables  or  functions 
within  the  code  itself. 

1)  Create  the  parameters  for  systems  A@  and  BY>£.  Using  Matlab,  theta,  gamma,  and 
epsilon  are  created  as  linearly  spaced  row  vectors.  Each  entry  in  the  vector 
corresponds  to  a  different  threshold  value  for  the  parameter.  At  the  max  threshold, 
the  accumulated  probability  is  approximately  one  and  for  the  minimum  threshold, 
the  accumulated  probability  is  approximately  zero. 

2)  Using  the  cumulative  density  function  (CDF)  for  the  normal  distribution,  the  true 
positive  and  false  positive  rates  are  calculated  for  individual  systems  Ae  and  BY>£. 
For  each  threshold,  a  different  probability  measurement  is  assigned.  System  BY5£  is 
created  in  such  a  way  that  the  convex  hull  of  the  ROC  curve  must  be  found 
whereas  classifier  Ae  naturally  produces  a  proper  ROC  curve.  Refer  to  the 
function  rocB  for  an  explanation  as  to  how  the  frontier  of  the  ROC  curve  for 
system  BY,£  is  created. 
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3)  Feeding  these  values  into  the  function  roc  computes  and  returns  the  combined 
Boolean  AND  ROC  curve  for  systems  Ae  and  Byx. 

4)  Using  the  same  information  as  generated  in  steps  1)  and  2),  inputting  this  data 
into  the  function  rocor  returns  the  combined  Boolean  OR  ROC  curve. 

C.1.2  Across  II  Simulation 

One  interesting  thing  to  note  at  this  time  is  the  following:  regardless  of  the 
constitution  of  the  targets  that  systems  Ae  and  Byx  are  designed  to  classify,  the 
performance  of  the  system  is  only  based  on  two  variables:  target/non- target  mean  and 
target/non-target  prevalence.  Hence,  the  function  for  the  true  positive  rate  for  system  A@ 
under  across  I  label  fusion  is  identical  to  the  true  positive  rate  for  system  AH  under  across 
II  label  fusion  assuming  the  target  prevalence  and  target  means  are  the  same.  This  same 
arguments  holds  for  classifier  system  Byx  as  well  as  across  III  and  within  label  fusion 
rules.  A  brief  outline  of  the  across  II  simulation  is  now  provided. 

1)  Load  the  same  parameters  (theta,  gamma,  epsilon)  and  ROC  curves  for 
classification  systems  Ae  and  Byx  that  were  used  for  the  combined  across  I 
simulation.  The  same  data  must  be  used  in  order  to  compare  the  performance  of 
systems  under  different  label-fusion  assumptions. 

2)  Use  the  true  positive  and  false  positive  rates  from  the  ROC  curves  of  classifier 
systems  A@  and  Byx  as  input  for  the  functions  a2roc/a2rocor  that  returns  the 
combined  across  II  AND  and  across  II  OR  ROC  curves  for  specified  levels  of 
performance  for  systems  Ae  and  Byy;. 
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The  algorithm  remains  approximately  the  same,  but  the  weighting  on  the  cost 
functions  for  classifier  A@  and  Byx  are  different  as  is  outlined  in  section  III. 

C.  1.3  -  Across  III  Simulation 

As  discussed  in  section  3,  in  an  across  III  label  fusion  environment,  classifier 
system  Ae  is  tuned  to  classify  the  set  union  of  ti  and  t2  and  classifier  system  Byx  is  tuned 
to  classify  the  set  union  of  t2  and  t3.  As  the  targets  for  systems  Ae  and  Byx  are  set  unions, 
the  calculation  of  true  positive  and  false  positive  rates  remain  the  same  as  the  function 
depends  only  on  parameter  setting  and  target  population  mean.  Recalling  the  cost 
functions  outlined  in  section  3,  the  cost  function  for  classifier  A@  is  weighted  by  the  prior 
probability  of  observing  an  inclusive  ti  or  t2  event  and  the  prior  probability  of  observing  a 
non-target  or  t3  event.  The  cost  function  for  classifier  Byx  is  weighted  by  the  prior 
probability  of  observing  either  a  t2  or  t3  event  and  the  prior  probability  of  observing  a 
non-target  or  ti  event. 

The  following  encompasses  a  brief  outline  of  the  across  III  algorithm. 

1)  Load  the  same  parameters  (theta,  gamma,  epsilon)  and  ROC  curves  for  classifiers 
Ae  and  Byx  that  were  used  for  the  combined  across  I  simulation.  The  same  data 
must  be  used  in  order  to  compare  the  performance  of  systems  under  different 
assumptions. 

2)  Load  this  information  into  the  roc  function  to  produce  the  ROC  curves  for  the 
combined  AND  system  and  the  rocor  function  for  the  combined  OR  ROC 
curve.  The  same  function  is  used  for  across  III  as  the  input  structure  is  identical  to 
that  of  across  I  except  that  the  prior  probability  weightings  on  the  true  positive 
and  false  positive  rates  for  classifiers  Ae  and  Byx  is  different. 
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The  only  difference  in  algorithm  between  across  I  and  across  III  lies  in  the  weighting  of 
target/non-targets  events  as  determined  by  the  cost  functions. 

C.1.4  Within  Simulation 

Recall  from  section  3  that  under  the  within  label  fusion  environment,  systems  Ae 
and  Bn  are  tuned  to  the  same  target  features.  In  this  case,  there  is  no  distinction  between 
target  types.  Interestingly,  the  calculations  for  the  true  positive  and  false  positive  rates  of 
the  individual  systems  is  still  the  same  due  to  the  function  relying  only  on  parameter 
setting  and  target/non-target  mean.  The  cost  functions  for  the  within  AND/OR  ROC 
curves  is  different  from  any  of  the  across  cost  functions  in  that  the  cost  function  for  each 
classifier  is  weighted  only  by  total  prior  target  prevalence  and  total  prior  non-target 
prevalence.  Because  of  this,  the  performance  of  the  combined  within  combined 
classification  system  is  considerably  better  than  the  performance  of  any  combined  across 
classification  system.  Most,  if  not  all,  classification  schemes  popular  in  literature  assume 
that  all  targets  and  non-targets  share  the  same  features  and  can  be  grouped  into  the  within 
hierarchy  of  classifiers  (N.J.  Leap  et  al:  2008). 

Below  is  a  short  description  of  the  within  algorithm. 

1)  Load  the  same  parameters  (theta,  gamma,  epsilon)  and  ROC  curves  for 
classification  systems  Ae  and  Bn  that  were  used  in  previous  simulations.  The 
same  data  must  be  used  in  order  to  compare  the  performance  of  systems  under 
different  assumptions. 

2)  Load  this  information  into  thewroc/wrocor  functions  to  produce  the  ROC 
curves  for  the  combined  AND/OR  systems. 


250 


C.2  Source  Code 


The  code  is  presented  here  and  any  functions  created  by  the  author  are  explained 
following  the  main  body  of  the  code.  Those  items  written  in  green  are  comments  issued 
by  the  author  that  will  assist  in  understanding  the  purpose  of  the  code. 

%%  Classification  systems  Ae  and  BYfS  are  created  here.  Note  that  the 
round  function  is  called  to  halt  the  level  of  computational  precision. 

theta=linspace (-4, 6, 301) ; 
gamma=theta; 

epsilon=linspace (0, 10, 301) ; 
n=length (theta) ; 

save ( ' parameters  1 ,  ' theta ' ,  ' gamma ' ,  ' epsilon ' ,  ' n ' )  ; 
for  i=l : n 

tpag (i) =round (probA (theta (i)  ,  0. 95)  *10000)  /10000; 
tpaf (i) =round (probA (theta (i) ,0.85) *10000) /10000; 
tpap (i) =round (probA (theta (i) ,0.75) *10000) /10000; 
fpa (i) =round (l/2*erfc (theta (i) /sqrt (2) ) *10000)  /10000; 

end 

save ( ' Af ront f ,  ' tpag ' ,  ' tpaf 1 ,  ' tpap ' ,  ' fpA f )  ; 

for  i=l : n 

for  j  =1 : n 

tpBg (i,  j  )  =round (probB (gamma (i) , epsilon (j ) ,0.95) *10000) /10000; 
tpBf (i, j ) =round (probB (gamma (i) , epsilon (j ) ,0.85) *10000) /10000; 
tpBp (i, j ) =round (probB (gamma (i) , epsilon (j ) ,0.75) *10000) /10000; 
fpB (i, j ) =round (1/2* (erf ( (gamma (i) +epsilon ( j ) ) /sqrt (2)  )  - 
erf (gamma (i) / sqrt (2) ) ) *10000) /10000; 
end 

end 

%  B  frontier 

ubg=rocB (tpBg, fpB) ; 

ubf=rocB ( tpBf , fpB) ; 

ubp=rocB ( tpBp , fpB) ; 

save ( ’ Bf ront ’  ,  ’ ubg ’ ,  ’ ubf ' ,  ' ubp ' ) 

%%  Within  AND/OR  all  prevalence 

%%  A  few  things  to  note.  First,  you  may  read  the  variable  "wgcl"  as 
"Within  combined  AND  ROC  curve  A =good  performance,  B=good  performance, 
correlation  P1  =  -0.8  (as  outlined  in  section  V)  .  Likewise,  "wfgcl"  is 
read  "Within  combined  AND  ROC  curve  A =fair  performance,  B=poor 
performance,  and  correlation  Px  =  -0.8.  If  a  c  is  not  specified  with  a 
variable,  this  implies  that  P  =  0  (such  as  in  the  case  where  i>3  & 
i<=4) . 


tic  %  begins  computation  timer, 
load ( 1 Af ront ' ) 
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load ( 1 Bf ront !  ) 


fpbg=ubg ( : , 1)  ; 
tpbg=ubg (: ,2) ; 
fpbf =ubf ( : , 1)  ; 
tpbf =ubf (:,2) ; 
f pbp=ubp (:,1) ; 
tpbp=ubp ( :  ,  2 ) ; 

i=l  ; 

while  i<=7 
if  i<=l 

load ( ' tenhancedeq '  )  %  Loads  the  prior  probabilities  for  the  target 

enhanced  environment 

wgcl=wroc (pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
wf cl=wroc (pet, pen, i , fpa, tpaf , fpbf, tpbf) ; 
wpcl=wroc (pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
wgf cl=wroc (pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
wgpcl=wroc (pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
wf gcl=wroc (pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
wfpcl=wroc (pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
wpgcl=wroc (pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
wpf cl=wroc (pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ’ C : \users\owner\desktop\within 

f iles\wdata\wencl ' , 'wgcl ' , '  wfcl ' , 'wpcl ' , 'wgfcl ' , 'wgpcl ' , 'wfgcl ' , 'wfpcl ' 
, ' wpgcl ' , ' wpf cl ' )  %  The  save  destination  is  arbitrary. 

wgorcl=wrocor (pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
wf orcl=wrocor (pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
wporcl=wrocor (pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
wgf orcl =wrocor (pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
wgporcl=wrocor (pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
wf gorcl=wrocor (pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
wfporcl=wrocor (pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
wpgorcl=wrocor (pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
wpf orcl =wrocor (pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ’ C : \users\owner\desktop\within 

f iles\wdata\wenclor ' , 'wgorcl', 'wforcl ' , 'wporcl', 'wgforcl', 'wgporcl', 'wf 
gorcl '  ,  'wfporcl ' ,  'wpgorcl ' ,  'wpforcl ' ) 

%  Target  rich  environment 

load ( ' tricheq ' )  %  Loads  the  prior  probabilities  for  the  target  rich 

environment 

wgrcl=wroc (pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
wf rcl=wroc (pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
wprcl=wroc (pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
wgf rcl=wroc (pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
wgprcl=wroc (pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
wf grcl=wroc (pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
wfprcl=wroc (pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
wpgrcl=wroc (pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
wpf rcl=wroc (pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
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save ( ' C : \users\owner\desktop\within 

f iles\wdata\wrichcl ’ ,  'wgrcl ’ ,  ' wfrcl ’ ,  ' wprcl ' ,  ' wgfrcl ' ,  'wgprcl ' ,  'wfgrcl ' 
,  'wfprcl ’ ,  'wpgrcl 1 ,  'wpfrcl ’ ) 

wgorrcl=wrocor (pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
wf orrcl=wrocor (pet, pen, i , fpa, tpaf , fpbf , tpbf ) ; 
wporrcl=wrocor (pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
wgf orrcl=wrocor (pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
wgporrcl=wrocor (pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
wf gorrcl=wrocor (pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
wfporrcl=wrocor (pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
wpgorrcl=wrocor (pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
wpf orrcl=wrocor (pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' C : \users\owner\desktop\within 

f iles\wdata\wrichclor '  , 'wgorrcl ' , 'wforrcl ' , 'wporrcl ' , 'wgforrcl', 'wgporr 
cl ' , ' wfgorrcl 1 , ' wfporrcl ’ , 'wpgorrcl 1 , ' wpforrcl ' ) 

%  Target  deficient  environment 

load ( ' tdef eq ' )  %Loads  the  prior  probabilities  for  the  target  deficient 
environment 

wgdcl=wroc (pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
wf dcl=wroc (pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
wpdcl=wroc (pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
wgf dcl=wroc (pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
wgpdcl=wroc (pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
wf gdcl=wroc (pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
wfpdcl=wroc (pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
wpgdcl=wroc (pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
wpf dcl=wroc (pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' C : \users\owner\desktop\within 

f iles\wdata\wdef cl ’ , ' wgdcl ’ , ' wfdcl ' , 'wpdcl ' , ’wgfdcl ’ , 'wgpdcl ' , ’wfgdcl ' , 
'wfpdcl ' , 'wpgdcl ' , ’wpf del ! ) 

wgordcl=wrocor (pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
wf ordcl=wrocor (pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
wpordcl=wrocor (pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
wgf ordcl=wrocor (pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
wgpordcl=wrocor (pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
wf gordcl=wrocor (pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
wfpordcl=wrocor (pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
wpgordcl=wrocor (pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
wpf ordcl=wrocor (pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' C : \users\owner\desktop\within 

f iles\wdata\wdef clor ’ , 'wgordcl', 'wfordcl', 'wpordcl', 'wgfordcl’, ’wgpordc 

1 ' , ’wfgordcl ’ , 'wfpordcl ’ , 'wpgordcl ' , 'wpfordcl ’ ) 

end 

if  i>l  &&  i<=2 
load ( ’ tenhancedeq' ) 

wgc2=wroc (pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
wf c2=wroc (pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
wpc2=wroc (pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
wgf c2=wroc (pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
wgpc2=wroc (pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
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wf gc2=wroc (pet, pen, i , fpa, tpaf , fpbg, tpbg) ; 
wfpc2=wroc (pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
wpgc2=wroc (pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
wpf c2=wroc (pet, pen, i , fpa, tpap, fpbf , tpbf ) ; 
save (  '  C : \users\owner\desktop\within 

f iles\wdata\wenc2 ' , 'wgc2 ' ,  fwfc2 ’ , 'wpc2 ' , 'wgfc2 ’ , 'wgpc2 ’ , 'wfgc2 ' , 'wfpc2 ' 
, ’ wpgc2 ’ , ’ wpf c2 ' ) 

wgorc2=wrocor (pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
wf orc2=wrocor (pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
wporc2=wrocor (pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
wgf orc2=wrocor (pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
wgporc2=wrocor (pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
wf gorc2=wrocor (pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
wfporc2=wrocor (pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
wpgorc2=wrocor (pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
wpf orc2=wrocor (pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save (  '  C : \users\owner\desktop\within 

files\wdata\wenc2or ' ,  'wgorc2',  'wforc2',  'wporc2',  'wgforc2',  'wgporc2',  'wf 
gorc2 '  ,  'wfporc2 ' ,  'wpgorc2 ' ,  'wpforc2 ' ) 

%  Target  rich  environment 
load ( ' tricheq ' ) 

wgrc2=wroc (pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
wf rc2=wroc (pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
wprc2=wroc (pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
wgf rc2=wroc (pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
wgprc2=wroc (pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
wf grc2=wroc (pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
wfprc2=wroc (pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
wpgrc2=wroc (pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
wpf rc2=wroc (pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' C : \users\owner\desktop\within 

f iles\wdata\wrichc2 ' , 'wgrc2 ' , 'wfrc2 ' , 'wprc2 ' , 'wgfrc2 ' , 'wgprc2 ' , 'wfgrc2 ' 
, 'wfprc2 ' , 'wpgrc2 ' , 'wpfrc2 ' ) 

wgorrc2=wrocor (pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
wf orrc2=wrocor (pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
wporrc2=wrocor (pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
wgf orrc2=wrocor (pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
wgporrc2=wrocor (pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
wf gorrc2=wrocor (pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
wfporrc2=wrocor (pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
wpgorrc2=wrocor (pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
wpf orrc2=wrocor (pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' C : \users\owner\desktop\within 

f iles\wdata\wrichc2or ' , 'wgorrc2', 'wforrc2 1 , 'wporrc2', 'wgforrc2', 'wgporr 
c2 ' ,  'wfgorrc2 '  ,  'wfporrc2 ' ,  'wpgorrc2 ' ,  'wpforrc2 ' ) 

%  Target  deficient  environment 
load ( ' tdef eq '  ) 

wgdc2=wroc (pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
wf dc2=wroc (pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
wpdc2=wroc (pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
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wgf dc2=wroc (pet, pen, i , fpa, tpag, fpbf , tpbf ) ; 
wgpdc2=wroc (pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
wf gdc2=wroc (pet, pen, i , fpa, tpaf , fpbg, tpbg) ; 
wfpdc2=wroc (pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
wpgdc2=wroc (pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
wpf dc2=wroc (pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' C : \users\owner\desktop\within 

f iles\wdata\wdef c2 ' , 'wgdc2 ' , 'wfdc2 ' , 'wpdc2 ' , 'wgfdc2 1 , 'wgpdc2 1 , 'wfgdc2 ' , 

' wfpdc2 1 , ' wpgdc2 1 , ' wpfdc2 ’ ) 

wgordc2=wrocor (pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
wf ordc2=wrocor (pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
wpordc2=wrocor (pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
wgf ordc2=wrocor (pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
wgpordc2=wrocor (pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
wf gordc2=wrocor (pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
wfpordc2=wrocor (pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
wpgordc2=wrocor (pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
wpf ordc2=wrocor (pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ’ C : \users\owner\desktop\within 

f iles\wdata\wdef c2or '  , 'wgordc2', 'wfordc2', 'wpordc2', 'wgfordc2', 'wgpordc 

2 ' , ' wfgordc2 ’ , ' wfpordc2 1 , ' wpgordc2 ' , ' wpfordc2 1 ) 

end 

%%  And  so  on  and  so  forth  up  through  i  =  7 . 

function  [Asys]  =  probA(x,y) 

%Probability  for  classifier  A 

%Ni  calculates  the  target/non-target  mean  of  the  system  by  the  taking 
the  norm  inverse  of  the  AUC  =  y  and  multiplying  the  by  the  square  root 
of  2  . 

%Asys  returns  the  probability  of  a  true  positive  at  a  given  x  =  theta 
%parameter  minus  the  mean. 

%"erfc"  stands  for  "error  function  complement" 

Ni=norminv (y) *sqrt (2 ) ; 

Asys=l/2*erf c ( (x-Ni) /sqrt (2)  )  ; 
end 

function  [Bsys]  =  probB(x,y,z) 

%Finding  TP/FP  value  for  classifier  B 

%Ni  is  short  for  norm  inverse  of  a  given  AUC  value 

%Bsys  approximates  the  solution  to  the  Gaussian  integral  for  bounds 
gamma 

%to  gamma  plus  epsilon 

%"erf"  stands  for  "error  function" 

Ni=norminv ( z ) *sqrt (2 )  ; 

Bsys=l/2* (erf ( ( (x+y) -Ni) / sqrt (2 ) ) -erf ( (x-Ni) / sqrt (2 ) ) ) ; 
end 

function  [rocB]  =  rocB(fp,tp) 

%  This  function  is  used  to  find  the  ROC  curve  for  classification  system 
%B. First,  input  variable  "fp"  =  false  positive  rates  and  input  variable 
%"tp"  =  true  positive  rates.  Take  these  NxN  matrices  and  reshape  them 
%into  (NA2)xl  column  vectors  and  then  sort  by  the  false  positive 
%values .  Take  the  difference  of  each  (i,j)  -  ( ( i  —  1 )  ,  ( j  — 1 )  )  entry  in  the 
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%true  postive  col  Instead  of  looping  over  all  values,  this  can  be 
%represented  as  a  matrix  operation  by  setting  the  first  entry  of  the 
%"dif f erence  matrix"  to  zero  (therefore  keeping  the  (1,1)  entry)  and 
%shifting  all  of  the  remaining  entries  down  by  1.  Take  the  difference 
%of  the  original  TP  col  with  this  column.  Now,  find  the  indices  where 
%the  difference  (i,j)  -  ( ( i  —  1 ) ,  ( j  — 1 ) )  <  0.  These  values  are  replaced  by 

%the  preceding  entry  so  that  the  true  positive  rate  is  always 
%increasing.  Take  the  unique  rows.  The  function  "rocheck"  determines 
%whether  or  not  there  are  any  remaining  (i,j)  -  ( ( i  —  1 )  ,  ( j  — 1 )  )  <  0,  if 
%so,  run  lines  17-33  again  until  rocheck  returns  the  logical  "1"  = 
%proper  ROC  curve.  Starting  at  vfp,  this  part  of  function  chooses  the 
%max  tpr  for  each  fpr  (useful  if  length ( rocB) is  computationally 
%unfeasible) . 

roc=s or trows ( cat ( 2 , fp ( : ) , tp ( : ) )  ,  1 )  ; 

c2=roc ( : , 2 ) / 

q=length (c2 ) ; 

dmat= [ 0 ; c2 (1:  (q-1) )  ]  ; 

dif f=c2-dmat ; 

cl=roc (  :  ,  1 )  ; 
nf=f ind (dif f<0 ) ; 
n=length (nf ) ; 

i=l; 

while  i<=n 

roc  (nf (i) ,  : ) =cat (2, cl (nf (i) ) ,c2 (nf (i) -1) ) ; 
i— i+1 ; 

end 

rocB=unique (roc, ' rows ’ ) ; 
i=l  ; 

while  i<=length ( rocB) ; 
if  rocheck ( rocB) ~=1 ; 

rocB=cloop (rocB) ; 
else  if  rocheck ( rocB) ==1 ; 
break 

end 

end 
i=i+l ; 
end 

vfp=rocB  (  : , 1 )  ; 
vtp=rocB  (  : , 2 )  ; 

for  i=l : length (vfp) 

ind=find (vfp==vfp (i) ) ; 
tpmax (i) =max (vtp (ind) ) ; 
fpmin (i) =min (vfp (ind)  )  ; 

end 

rocB=unique ( cat ( 2 , fpmin ' , tpmax ’ ) , ’ rows ’ ) ; 
end 
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function  [wroc]  =  wroc (a,b,g,w,x,y, z) 
if  g==l 

rho=-0 . 8 ; 
else  if  g==2 

rho=-0 . 5 / 
else  if  g==3 

rho=-0 . 3 / 
else  if  g==4 

rho=0 ; 

else  if  g==5 

rho=0 . 3 ; 
else  if  g==6 

rho=0 . 5 ; 
else  if  g==7 

rho=0 . 8 ; 

end 

end 

end 

end 

end 

end 

end 

n=length (w) ; 
m=length (y) ; 

%  Cost  functions  for  within  along  with  correlation  term, 
for  i=l : n 

for  j  =1 : m 

fpand  (i,  j  )=w(i)*y(j); 
costA ( i ) =a*x ( i ) +b*w ( i ) ; 
costB ( j ) =a*z ( j ) +b*y ( j ) ; 

corr (i, j ) =sqrt (costA (i) * (1-cost A (i) ) ) *sqrt (costB (j ) * (1- 
costB ( j ) ) ) / 

tpand (i,j)  =  (l/a) * (rho*corr (if  j ) +costA (i) ^ costB ( j ) - 
b* fpand (i, j ) ) ; 
end 

end 

%  Under  certain  correlations,  the  max(tpr)  >  1,  this  set  the  upper 
%  bound  at  1 . 

improc=s or trows (cat (2, fpand ( : ) , tpand ( : ) ) , 1 ) ; 
c2=improc (: ,2) ; 
btp=f ind ( c2<0 ) ; 
if  isempty (btp) ~=1 ; 
c2 (btp) =0 ; 

end 

ind=f ind ( c2>l )  ; 
if  isempty ( ind) ~=1 ; 
c2 ( ind) =1 ; 
end 

improc=cat ( 2 , improc ( : , 1 )  ,  c2 )  ; 

c2=improc (: ,2) ; 
q=length (c2 ) ; 
dmat= [ 0 ; c2 ( 1 : (q-1))]; 
dif f=c2-dmat ; 
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cl=improc  ( :  ,  1 )  ; 
nf=f ind (dif f<0 ) ; 
n=length (nf ) ; 

i=l; 

while  i<=n 

improc (nf (i) , : ) =cat (2, cl (nf (i) ) , c2 (nf (i) -1) ) ; 
i=i+l ; 

end 

wroc=unique (round (improc* 10000) /10000,  ’ rows  f )  ; 
i=l  ; 

while  i<=length (wroc)  %usually  takes  about  3000  iterations  before 
following  step  can  assure  a  proper  ROC  curve  for  the  size  of  this  data 
set . 

if  rocheck (wroc) ~=1 ; 

wroc=cloop (wroc) ; 
else  if  rocheck (wroc) ==1 ; 
break 

end 

end 

i=i+l; 

end 

vfp=wroc  ( : , 1 )  ; 
vtp=wroc  (  : , 2 )  ; 

for  i=l : length (vfp) 

ind=find (vfp==vfp (i) ) ; 
tpmax (i) =max (vtp (ind) ) ; 
fpmin (i) =min (vfp (ind) ) ; 

end 

wroc=unique ( cat ( 2 ,  fpmin ' ,  tpmax ’ ) ,  ’ rows  f )  / 
end 


function  [wrocor]  =  wrocor (a,b,g,w,x,y, z) 

%  Almost  exactly  the  same  as  wroc,  but  cost  functions  are  different, 
if  g==l 

rho=-0 . 8 ; 
else  if  g==2 

rho=-0 . 5 ; 
else  if  g==3 

rho=-0 . 3 ; 
else  if  g==4 

rho=0 ; 

else  if  g==5 

rho=0 . 3 ; 
else  if  g==6 

rho=0 . 5 ; 
else  if  g==7 
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rho=0 . 8 ; 


end 

end 

end 

end 

end 

end 

end 

n=length (w) ; 
m=length (y) ; 
for  i=l : n 

for  j  =1 : m 

fpor  (i,j)=w(i)+y(j)-w(i)*y(j); 
costA ( i ) =a*x ( i ) +b*w ( i ) ; 
costB ( j ) =a*  z ( j ) +b*y ( j ) ; 

corr (i, j ) =sqrt (costA (i) * (1-cost A (i) ) ) *sqrt (costB (j ) * (1- 
costB ( j ) ) ) / 

tpor  (i,  j )  =  (1/a) * (costA(i)+costB(j) -rho*corr (i,  j  )  - 
(costA (i) * costB ( j ) ) -b*fpor (i, j ) ) ; 
end 

end 

improc=s or trows ( cat ( 2 , fpor ( : ) , tpor (:)) ,1) ; 
c2=improc (: ,2) ; 
btp=f ind (c2<0 ) ; 
if  isempty (btp) ~=1 ; 
c2 (btp) =0 ; 

end 

ind=f ind ( c2>l )  ; 
if  isempty ( ind) ~=1 ; 
c2 ( ind) =1 ; 
end 

improc=cat ( 2 , improc ( : , 1 )  ,  c2 )  ; 

c2=improc (: ,2) ; 
q=length (c2 ) ; 
dmat= [ 0 ; c2 ( 1 : (q-1))]; 
dif f=c2-dmat ; 

cl=improc  (  : , 1 )  ; 
nf=f ind (dif f<0 ) ; 
n=length (nf ) ; 

i=l; 

while  i<=n 

improc (nf (i) , : ) =cat (2, cl (nf (i) ) ,c2 (nf (i) -1) ) ; 
i=i+l ; 

end 

wrocor=unique (round (improc^ 10 000) /10000f  rows ' ) ; 
i=l  ; 

while  i<=2000 
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if  rocheck (wrocor ) ~=1 ; 

wrocor=cloop (wrocor ) ; 
else  if  rocheck (wrocor) ==1 ; 
break 

end 

end 
i=i+l ; 
end 

vfp=wrocor  (:,1)  ; 
vtp=wrocor  (:,2)  ; 

for  i=l : length (vfp) 

ind=find (vfp==vfp (i) ) ; 
tpmax (i) =max (vtp (ind) )  ; 
fpmin (i) =min (vfp (ind) ) ; 

end 

wrocor=unique ( cat ( 2 , fpmin ' ,  tpmax ’ ) ,  ' rows ' ) ; 
end 

function  [cloop]  =  cloop (x) 

%cloop  performs  the  same  "difference"  function  as  outlined  in  all  roc 
%f unctions 

cloop=cat (2 , x ( : , 1 ) ,x(:,2) ) ; 

cl=cloop ( : , 1 ) ; 

c2=cloop ( : , 2 ) ; 

q=length (cloop) ; 

dmat= [ 0 ; c2 ( 1 : (q-1))]; 

diff=c2-dmat; 

nf=f ind (dif f<0 ) ; 
n=length (nf ) ; 

i=l; 

while  i<=n 

cloop  (nf (i) ,  : ) =cat (2, cl (nf (i) ) ,c2 (nf (i)-l) ) ; 
i=i+l ; 

end 

cloop=unique (round (cloop *10000) /10000, ’ rows ’ ) ; 
end 

function  [rocheck]  =  rocheck (x) 

%Checks  to  make  sure  that  all  true  positive  rates  are  strictly 
increasing . 

%It  returns  a  "1"  if  this  condition  is  true  and  a  "0"  if  this  condition 
is 

%f alse . 
q=length (x); 
c2=x  (  : , 2 )  ; 

dmat= [ 0 ; c2  (1 :  (q-1))]; 
dif f=c2-dmat ; 

if  dif f >=0 ; 

rocheck=l ; 


260 


else 


rocheck=0 ; 

end 

end 

%  Across  II  all  prev 
tic 

load ( ’ Af ront ’ ) 
load ( ’ Bf ront ’  ) 

%  Target  enhanced  environment 

fpbg=ubg (:,1) ; 

tpbg=ubg ( : , 2 ) ; 

fpbf=ubf ( : , 1) ; 

tpbf =ubf ( : , 2 ) ; 

f pbp=ubp ( :  ,  1 )  ; 

tpbp=ubp ( : , 2 ) ; 

1=1; 

while  i<=7 
if  i<=l 
load ( ’ tena2 ' ) 

a2gcl=roc2 (pet A, penA, pet, pen, i, fpa, tpag, fpbg, tpbg) ; 
a2f cl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf , fpbf , tpbf) ; 
a2pcl=roc2 (pet A, penA, pet, pen, i, fpa, tpap, fpbp, tpbp) ; 
a2gf cl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
a2gpcl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
a2fgcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf, fpbg, tpbg) ; 
a2fpcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf, fpbp, tpbp) ; 
a2pgcl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
a2pf cl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' c : \users\owner\desktop\a2 

f iles\a2data\a2encl '  ,  fa2gcl', 'a2fcl', 'a2pcl', ,a2gfclf, 'a2gpcl!,  !a2fgcl' 
, ’ a2fpcl ' , ' a2pgcl ' , ’ a2pfcl ' ) 

a2gorcl=roc2or (pet A, penA, pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
a2f orcl=roc2or (pet A, penA, pet, pen, i, fpa, tpaf, fpbf, tpbf) ; 
a2porcl=roc2or (pet A, penA, pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
a2gf orcl=roc2or (pet A, penA, pet , pen, i , fpa, tpag, fpbf, tpbf) ; 
a2gporcl=roc2or (pet A, penA, pet , pen, i , fpa, tpag, fpbp, tpbp) ; 
a2 f gorcl=roc2or (pet A, penA, pet, pen, i, fpa, tpaf, fpbg, tpbg) ; 
a2fporcl=roc2or (pet A, penA, pet, pen, i, fpa, tpaf, fpbp, tpbp) ; 
a2pgorcl=roc2or (pet A, penA, pet , pen, i , fpa, tpap, fpbg, tpbg) ; 
a2pf orcl=roc2or (pet A, penA, pet , pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' c : \users\owner\desktop\a2 

f iles\a2data\a2enclor ' , 'a2gorcl', ,a2forcl', 'a2porcl', ,a2gforcl', 'a2gpor 
cl ' ,  'a2fgorcl 1  ,  'a2 fporcl ' ,  ! a2pgorcl ',  1 a2pf orcl 1 ) 

%  Target  rich  environment 
load ( ’ tricha2 1  ) 

a2grcl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
a2f rcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf, fpbf, tpbf) ; 
a2prcl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
a2gf rcl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
a2gprcl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
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a2fgrcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf , fpbg, tpbg) ; 
a2fprcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf, fpbp, tpbp) ; 
a2pgrcl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
a2pf rcl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbf , tpbf ) ; 
save ( 1 c : \users\owner\desktop\a2 

f iles\a2data\a2richcl '  , 'a2grcl', 'a2frcl', 'a2prcl', 'a2gfrcl', 'a2gprcl', 1 
a2fgrcl ’  ,  ' a2  fprcl ’ ,  ’ a2pgrcl ’ ,  ' a2pf rcl ' ) 

a2gorrcl=roc2or (pet A, penA, pet , pen, i , fpa, tpag, fpbg, tpbg) ; 
a2 f orrcl=roc2or (pet A, penA, pet, pen, i, fpa, tpaf, fpbf, tpbf) ; 
a2porrcl=roc2or (pet A, penA, pet , pen, i , fpa, tpap, fpbp, tpbp) ; 
a2gf orrcl=roc2or (pet A, penA, pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
a2gporrcl=roc2or (pet A, penA, pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
a2  f gorrcl=roc2or (pet A, penA, pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
a2  fporrcl=roc2or (pet A, penA, pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
a2pgorrcl=roc2or (pet A, penA, pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
a2pf orrcl=roc2or (pet A, penA, pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' c : \users\owner\desktop\a2 

f iles\a2data\a2richclor ! ,  'a2gorrcl',  'a2forrcl',  'a2porrcl',  'a2gforrcl!,  1 
a2gporrcl 1 ,  ' a2fgorrcl 1  ,  ' a2fporrcl ' ,  ’ a2pgorrcl 1 ,  ' a2pforrcl ’ ) 

%  Target  deficient  environment 
load ( ’ tdef a2 '  ) 

a2gdcl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
a2fdcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf, fpbf, tpbf) ; 
a2pdcl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
a2gf dcl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
a2gpdcl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
a2fgdcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf, fpbg, tpbg) ; 
a2fpdcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf, fpbp, tpbp) ; 
a2pgdcl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
a2pf dcl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' c : \ user s\ owner \desktop\a2 

f iles\a2data\a2def cl ' ,  ' a2gdcl '  ,  ' a2fdcl '  ,  ' a2pdcl ' ,  ' a2gfdcl '  ,  ' a2gpdcl ’ ,  ' a 
2fgdcl '  ,  ' a2fpdcl ' ,  ' a2pgdcl ’ ,  1 a2pfdcl ! ) 

a2gordcl=roc2or (pet A, penA, pet , pen, i , fpa, tpag, fpbg, tpbg) ; 
a2f ordcl=roc2or (pet A, penA, pet, pen, i, fpa, tpaf, fpbf, tpbf) ; 
a2pordcl=roc2or (pet A, penA, pet , pen, i , fpa, tpap, fpbp, tpbp) ; 
a2gf ordcl=roc2or (pet A, penA, pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
a2gpordcl=roc2or (pet A, penA, pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
a2fgordcl=roc2or (pet A, penA, pet, pen, i, fpa, tpaf, fpbg, tpbg) ; 
a2fpordcl=roc2or (pet A, penA, pet, pen, i, fpa, tpaf, fpbp, tpbp) ; 
a2pgordcl=roc2or (pet A, penA, pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
a2pf ordcl=roc2or (pet A, penA, pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' c : \ user s\ owner \desktop\a2 

f iles\a2data\a2def clor ' , ,a2gordcl’, ,a2fordclf, 'a2pordcl', 'a2gfordcl', 'a 

2gpordcl ' ,  ’ a2fgordcl ' ,  ' a2fpordcl ’ ,  ’ a2pgordcl '  ,  ’ a2pfordcl ’ ) 

end 

%  We  will  abbreviate  this  here  as  it  just  continues  on  through  the 
remainder  of  the  possible  values  of  the  correlation  coefficient. 

%  Across  II  all  prev  unequal  target  priors.  In  much  the  same  spirit  as 
above,  this  is  an  example  of  the  code  where  we  are  solving  for  the  ROC 
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curves  after  altering  the  prevalence  of  targets  in  the  Across  II 
environment . 

tic 

load ( ’ Af ront ’ ) 
load ( ’ Bf ront ’ ) 

%  Target  enhanced  environment 

fpbg=ubg ( : , 1)  ; 

tpbg=ubg (: ,2) ; 

fpbf =ubf ( : , 1)  ; 

tpbf =ubf ( : , 2 ) ; 

f pbp=ubp ( : , 1 ) ; 

tpbp=ubp (: ,2) ; 

i=l  ; 

while  i<=7 
if  i<=l 

load ( 1 altpreva2 ’ ) 

aa2gcl=roc2 (pet A,  penA, pet , pen, i ,  fpa,  tpag, fpbg,  tpbg) ; 
aa2f cl=roc2 (pet A,  penA, pet,  pen, i, fpa, tpaf , fpbf, tpbf) ; 
aa2pcl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
aa2gf cl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
aa2gpcl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
aa2fgcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf, fpbg, tpbg) ; 
aa2fpcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf, fpbp, tpbp) ; 
aa2pgcl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
aa2pf cl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ’ c : \ user s\ owner \desktop\a2 

f iles\a2data\aa2encl ' , 'aa2gcl', 'aa2fcl', 'aa2pcl', 'aa2gfcl', 'aa2gpcl', 'a 
a2fgcl '  ,  ' aa2fpcl ' ,  ' aa2pgcl ' ,  ' aa2pfcl ' ) 

aa2gorcl=roc2or (pet A, penA, pet , pen, i , fpa, tpag, fpbg, tpbg) ; 
aa2f orcl=roc2or (pet A, penA, pet, pen, i, fpa, tpaf, fpbf, tpbf) ; 
aa2porcl=roc2or (pet A, penA, pet , pen, i , fpa, tpap, fpbp, tpbp) ; 
aa2gf orcl=roc2or (pet A, penA, pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
aa2gporcl=roc2or (pet A, penA, pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
aa2fgorcl=roc2or (pet A, penA, pet, pen, i, fpa, tpaf, fpbg, tpbg) ; 
aa2fporcl=roc2or (pet A, penA, pet, pen, i, fpa, tpaf, fpbp, tpbp) ; 
aa2pgorcl=roc2or (pet A, penA, pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
aa2pf orcl=roc2or (pet A, penA, pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' c : \ user s\ owner \desktop\a2 

f iles\a2data\aa2enclor ’  ,  ,aa2gorcl’,  ,aa2forcl',  ,aa2porcl’,  ,aa2gforcl',  'a 
a2gporcl ’ , ' aa2fgorcl ' , ’ aa2fporcl ’ , ' aa2pgorcl ' , ' aa2pforcl ’ ) 

%  Target  rich  environment 
load ( ' apricha2 ’ ) 

aa2grcl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
aa2f rcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf, fpbf, tpbf) ; 
aa2prcl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
aa2gf rcl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
aa2gprcl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
aa2 f grcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf, fpbg, tpbg) ; 
aa2 fprcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf, fpbp, tpbp) ; 
aa2pgrcl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
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aa2pf rcl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbf , tpbf ) ; 
save ( ’ c : \users\owner\desktop\a2 

f iles\a2data\aa2richcl '  , 'aa2grcl', 'aa2frcl', 'aa2prcl', 'aa2gfrcl', 'aa2gp 
rcl ' , ' aa2fgrcl ' , ' aa2fprcl ' , ' aa2pgrcl ' , ' aa2pfrcl ! ) 

aa2gorrcl=roc2or (pet A, penA, pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
aa2f orrcl=roc2or (pet A, penA, pet, pen, i, fpa, tpaf , fpbf, tpbf) ; 
aa2porrcl=roc2or (pet A, penA, pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
aa2gf orrcl=roc2or (pet A, penA, pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
aa2gporrcl=roc2or (pet A, penA, pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
aa2fgorrcl=roc2or (pet A, penA, pet, pen, i, fpa, tpaf, fpbg, tpbg) ; 
aa2fporrcl=roc2or (pet A, penA, pet, pen, i, fpa, tpaf, fpbp, tpbp) ; 
aa2pgorrcl=roc2or (pet A, penA, pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
aa2pf orrcl=roc2or (pet A, penA, pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' c : \users\owner\desktop\a2 

f iles\a2data\aa2richclor '  , 'aa2gorrcl', 'aa2forrcl', 'aa2porrcl', 'aa2gforr 
cl 1 ,  1 aa2gporrcl ' ,  1 aa2fgorrcl 1  ,  1 aa2fporrcl 1 ,  ' aa2pgorrcl 1 ,  1 aa2pforrcl ' ) 

%  Target  deficient  environment 
load ( ’ apdef a2 ' ) 

aa2gdcl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
aa2fdcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf, fpbf, tpbf) ; 
aa2pdcl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
aa2gf dcl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
aa2gpdcl=roc2 (pet A, penA, pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
aa2 f gdcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf, fpbg, tpbg) ; 
aa2 fpdcl=roc2 (pet A, penA, pet, pen, i, fpa, tpaf, fpbp, tpbp) ; 
aa2pgdcl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
aa2pf dcl=roc2 (pet A, penA, pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ’ c : \users\ owner \desktop\a2 

f iles\a2data\aa2def cl ' , ' aa2gdcl ' , ' aa2fdcl ' , ' aa2pdcl ' , ' aa2gfdcl 1 , ' aa2gpd 
cl ’ ,  ' aa2fgdcl ’  ,  ’ aa2fpdcl ' ,  ' aa2pgdcl ’ ,  ' aa2pfdcl ' ) 

aa2gordcl=roc2or (pet A, penA, pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
aa2  f ordcl=roc2or (pet A, penA, pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
aa2pordcl=roc2or (pet A, penA, pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
aa2gf ordcl=roc2or (pet A, penA, pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
aa2gpordcl=roc2or (pet A, penA, pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
aa2 f gordcl=roc2or (pet A, penA, pet, pen, i, fpa, tpaf, fpbg, tpbg) ; 
aa2 fpordcl=roc2or (pet A, penA, pet, pen, i, fpa, tpaf, fpbp, tpbp) ; 
aa2pgordcl=roc2or (pet A, penA, pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
aa2pf ordcl=roc2or (pet A, penA, pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' c : \users\owner\desktop\a2 

f iles\a2data\aa2def clor ' , 'aa2gordcl', 'aa2fordcl', 'aa2pordcl', 'aa2gfordc 
1 ’ ,  ’ aa2gpordcl ' ,  ' aa2fgordcl ’  ,  ’ aa2fpordcl ’ ,  ’ aa2pgordcl ' ,  ’ aa2pfordcl ! ) 
end 

%  And  so  on... 

function  [roc2]  =  roc2 (a, b, c, d, g, w, x, y, z ) 

%  roc  function  for  across  II  AND 
if  g==l 

rho=-0 . 8 ; 
else  if  g==2 

rho=-0 . 5 ; 
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else  if  g==3 

rho=-0 . 3 ; 
else  if  g==4 

rho=0 ; 

else  if  g==5 

rho=0 . 3 ; 
else  if  g==6 

rho=0 . 5 ; 
else  if  g==7 

rho=0 . 8 ; 

end 

end 

end 

end 

end 

end 

end 

n=length (w) ; 
m=length (y) ; 
for  i=l : n 

for  j  =1 : m 

fpand  (i,  j  )=w(i)*y(j); 
costA ( i ) =a*x ( i ) +b*w ( i ) ; 
costB (j ) =c*z ( j ) +d*y ( j ) ; 

corr (i, j ) =sqrt (costA (i) * (1-cost A (i) ) ) *sqrt (costB ( j ) * (1- 
costB  ( j ) ) )  / 

tpand (i, j ) = (1/c) * (rho^corr (if j ) + costA ( i ) * costB ( j ) - 
d* fpand (i, j ) ) ; 
end 

end 

improc=s or trows (cat (2, fpand ( : ) ,  tpand ( : ) ) , 1 ) ; 
c2=improc (: ,2) ; 
btp=f ind ( c2<0 ) / 
if  isempty (btp) ~=1  ; 
c2 (btp) =0 ; 

end 

ind=f ind ( c2>l )  ; 
if  isempty ( ind) ~=1 ; 
c2 ( ind) =1 ; 
end 

improc=cat ( 2 , improc ( : , 1 )  ,  c2 )  ; 

c2=improc ( : , 2 ) ; 
q=length (c2 ) ; 
dmat= [ 0 ; c2 ( 1 : (q-1))]; 
dif f=c2-dmat ; 

cl=improc  ( : , 1 )  ; 
nf=f ind (dif f<0 ) ; 
n=length (nf ) ; 

i=l; 
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while  i<=n 

improc (nf ( i ) ,  : ) =cat (2, cl (nf (i) ) ,  c2 (nf (i) -1) ) 
i=i+l ; 

end 

roc2=unique (round (improc* 10000) /10000, ’ rows ’ 
i=l  ; 

while  i<=length ( roc2 ) 
if  rocheck ( roc2 ) ~=1 ; 

roc2=cloop ( roc2 ) ; 
else  if  rocheck ( roc2 ) ==1 ; 
break 

end 

end 
i=i+l ; 
end 

vfp=roc2  ( : , 1 )  ; 
vtp=roc2  (: ,2)  ; 

for  i=l : length (vfp) 

ind=f ind ( vfp==vfp ( i ) ) / 
tpmax (i) =max (vtp (ind) ) ; 
fpmin (i) =min (vfp (ind) ) ; 

end 

roc2=unique ( cat ( 2 ,  fpmin ’ ,  tpmax ’ ) , ’ rows ’ ) ; 
end 

function  [roc2or]  =  roc2or (a, b, c, d, q, w, x, y, z 
%  function  for  across  II  OR  ROC 
if  g==l 

rho=-0 . 8 ; 
else  if  g==2 

rho=-0 . 5 ; 
else  if  g==3 

rho=-0 . 3 / 
else  if  g==4 

rho=0 ; 

else  if  g==5 

rho=0 . 3 ; 
else  if  g==6 

rho=0 . 5 ; 
else  if  g==7 

rho=0 . 8 ; 

end 

end 

end 

end 

end 

end 

end 

n=length (w) ; 
m=length (y) ; 
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for  i=l : n 

for  j  =1 : m 

fpor  (i,  j ) =w (i) +y ( j ) -w (i) *y  ( j ) ; 
costA ( i ) =a*x ( i ) +b*w ( i ) ; 
costB  ( j ) =c*  z ( j ) +d*y ( j ) ; 

corr (i, j ) =sqrt (costA (i) * (1-cost A (i) ) ) *sqrt (costB (j ) * (1- 
costB  ( j ) ) )  / 

tpor (i,  j )  =  (1/c) * (costA(i)+costB(j) -rho*corr (i, j ) - 
(costA (i) * costB ( j ) ) -d*fpor (i, j ) ) ; 
end 

end 

improc=s or trows ( cat ( 2 , fpor ( : ) , tpor (  :  )  )  ,  1 )  ; 
c2=improc ( : , 2 ) ; 
btp=f ind ( c2<0 ) / 
if  isempty (btp) ~=1 ; 
c2 (btp) =0 ; 

end 

ind=f ind ( c2>l )  ; 
if  isempty ( ind) ~=1 ; 
c2 ( ind) =1 ; 
end 

improc=cat ( 2 , improc ( : , 1 )  ,  c2 )  ; 

c2=improc ( : , 2 ) ; 
q=length (c2 ) ; 
dmat= [ 0 ; c2 ( 1 : (q-1))]; 
dif f=c2-dmat ; 

cl=improc (:,1)  ; 
nf=f ind (dif f<0 ) ; 
n=length (nf ) ; 

i=l; 

while  i<=n 

improc (nf (i) , : ) =cat (2, cl (nf (i) ) , c2  (nf (i) -1) ) ; 
i=i+l ; 

end 

roc2or=unique (round (improc^ 10 000) /10000f  rows ' ) / 
i=l  ; 

while  i<=length ( roc2or ) ; 
if  rocheck ( roc2or )  ~=1  ; 

roc2or=cloop ( roc2or ) ; 
else  if  rocheck ( roc2or) ==1 ; 
break 

end 

end 
i=i+l ; 
end 

vfp=roc2or  (  : r  1)  ; 
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vtp=roc2or ( : ,2) ; 


for  i=l : length (vfp) 

ind=f ind ( vfp==vfp ( i ) ) ; 
tpmax (i) =max (vtp (ind) )  ; 
fpmin (i) =min (vfp (ind)  )  ; 

end 

roc2or=unique ( cat ( 2 ,  fpmin ’ ,  tpmax ’  )  ,  ' rows ’ ) ; 
end 

Note:  The  following  section  of  code  works  for  both  across  I  and  across  III,  only  the  input 
is  different.  This  general  algorithm  follows  the  same  outline  as  before. 

%  Across  III  all  prev 
tic 

load ( ’ Af ront ' ) 
load ( ’ Bf ront ' ) 

fpbg=ubg ( : , 1 ) ; 
tpbg=ubg ( : ,  2 ) ; 
fpbf =ubf ( :  ,  1 )  ; 
tpbf =ubf ( : ,  2 ) ; 
f pbp=ubp {:,!); 
tpbp=ubp (  :  ,  2  ) ; 

i=l; 

while  i<=7 
if  i<=l 
load ( ’ tena3 ! ) 

a3gcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
a3f cl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf , fpbf, tpbf) ; 
a3pcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
a3gf cl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
a3gpcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
a3f gcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
a3fpcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
a3pgcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
a3pf cl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' C : \users\owner\desktop\A3 

f iles\a3data\a3encl ' ,  ’a3gcl',  ’a3fcl’,  1 a3pclf,  'a3gfcl',  'a3gpcl',  'a3fgcl' 

, ’ a3fpcl ' , ' a3pgcl ’ , ' a3pf cl ’ ) 

a3gorcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
a3f orcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
a3porcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
a3gf orcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
a3gporcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
a3f gorcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
a3fporcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
a3pgorcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
a3pf orcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
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save (  '  C : \users\owner\desktop\A3 

f iles\a3data\a3enclor '  , 'a3gorcl', 'a3forcl', 'a3porcl',  'a3gforcl', 'a3gpor 
cl' ,  ' a3f gorcl '  ,  ' a3fporcl ' ,  ' a3pgorcl ' ,  ' a3pf orcl ' ) 

%  Target  rich  environment 
load ( ' tricha3 ' ) 

a3grcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
a3f rcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf , fpbf , tpbf ) ; 
a3prcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
a3gf rcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
a3gprcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
a3f grcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
a3fprcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
a3pgrcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
a3pf rcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' C : \users\owner\desktop\A3 

f iles\a3data\a3richcl '  ,  'a3grcl',  fa3frclf,  'a3prcl',  'a3gfrcl',  'a3gprcl',  ' 
a3fgrcl '  ,  ' a3fprcl ' ,  ' a3pgrcl ' ,  ' a3pf rcl ' ) 

a3gorrcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
a3f orrcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
a3porrcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
a3gf orrcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
a3gporrcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
a3f gorrcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
a3fporrcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
a3pgorrcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
a3pf orrcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' C : \users\owner\desktop\A3 

f iles\a3data\a3richclor ' , 'a3gorrcl', 'a3forrcl', 'a3porrcl', 'a3gforrcl', ' 
a3gporrcl ' ,  ' a3fgorrcl '  ,  ' a3fporrcl ' ,  ' a3pgorrcl ' ,  ' a3pforrcl ' ) 

%  Target  deficient  environment 
load ( ' tdefa3'  ) 

a3gdcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
a3f dcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
a3pdcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
a3gf dcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
a3gpdcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
a3f gdcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
a3fpdcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
a3pgdcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
a3pf dcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' C : \users\owner\desktop\A3 

f iles\a3data\a3def cl ' ,  ' a3gdcl ' ,  ' a3fdcl '  ,  ' a3pdcl ' ,  ' a3gfdcl '  ,  ' a3gpdcl ' ,  ' a 
3fgdcl ' , ' a3fpdcl ' , ' a3pgdcl ' , ' a3pfdcl ' ) 

a3gordcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
a3f ordcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
a3pordcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
a3gf ordcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
a3gpordcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
a3f gordcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
a3fpordcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
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a3pgordcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
a3p fordo l=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpap, fpbf , tpbf ) ; 
save ( ’ C : \users\owner\desktop\A3 

f iles\a3data\a3def clor ' ,  ' a3gordcl',  'a3fordcl',  ' a3pordcl',  'a3gfordcl',  ’a 

3gpordcl ’ , ' a3fgordcl ' , ’ a3fpordcl ’ , ' a3pgordcl ’ , ' a3pfordcl ' ) 

end 

%  Up  to  i=7  and  just  for  kicks  we'll  look  at  one  iteration  of  the  loop 
that  solves  for  those  ROC  curves  where  the  prevalence  of  t12  and  t23  are 
altered  under  the  across  III  label  fusion  rule. 

%  Across  III  all  prev 
tic 

load ( ’ Af ront ’ ) 
load ( ’ Bf ront ’  ) 

fpbg=ubg ( : , 1 ) ; 
tpbg=ubg ( : , 2 ) ; 
fpbf=ubf ( : , 1) ; 
tpbf =ubf ( : , 2 ) ; 
f pbp=ubp ( : , 1 ) ; 
tpbp=ubp ( : , 2 ) ; 

i=l  ; 

while  i<=7 
if  i<=l 

load ( ’ altpreva3 ’ ) 

aa3gcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
aa3f cl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpaf , fpbf, tpbf) ; 
aa3pcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
aa3gf cl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
aa3gpcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
aa3f gcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
aa3fpcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
aa3pgcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
aa3pf cl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ’ C : \users\owner\desktop\A3 

f iles\a3data\aa3encl ' ,  ' aa3gcl',  'aa3fcl',  ' aa3pcl',  'aa3gfcl',  ' aa3gpcl',  'a 
a3fgcl ’  ,  1 aa3fpcl 1 ,  1 aa3pgcl ’ ,  1 aa3pf cl ! ) 

aa3gorcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
aa3f orcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
aa3porcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
aa3gf orcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
aa3gporcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
aa3f gorcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
aa3fporcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
aa3pgorcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
aa3pf orcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ’ C : \users\owner\desktop\A3 

f iles\a3data\aa3enclor ’  ,  'aa3gorcl',  'aa3forcl',  'aa3porcl',  * aa3gforclf,  ’a 
a3gporcl ' , ’ aa3fgorcl ' , ' aa3fporcl ' , ’ aa3pgorcl ' , ’ aa3pforcl ’ ) 

%  Target  rich  environment 
load ( ’ apricha3 ' ) 
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aa3grcl=roc (pet 12 , pen 12  , pet23, pen23, pet, pen, i, fpa, tpag, fpbg, tpbg) ; 
aa3f rcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf , fpbf , tpbf ) ; 
aa3prcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
aa3gf rcl=roc (pet 12 , pen 12 , pet23, pen23, pet, pen, i, fpa, tpag, fpbf, tpbf) ; 
aa3gprcl=roc (pet 12 , pen 12 , pet23, pen23, pet, pen, i, fpa, tpag, fpbp, tpbp) ; 
aa3fgrcl=roc (pet 12 , pen 12 , pet23, pen23, pet, pen, i, fpa, tpaf, fpbg, tpbg) ; 
aa3fprcl=roc (pet 12 , pen 12 , pet23, pen23, pet, pen, i, fpa, tpaf, fpbp, tpbp) ; 
aa3pgrcl=roc (pet 12 , pen 12 , pet23, pen23, pet, pen, i, fpa, tpap, fpbg, tpbg) ; 
aa3pf rcl=roc (pet 12 , pen 12 , pet23, pen23, pet, pen, i, fpa, tpap, fpbf, tpbf) ; 
save ( ' C : \users\owner\desktop\A3 

f iles\a3data\aa3richcl ' , 'aa3grcl', 'aa3frcl', 'aa3prcl',  'aa3gfrcl', ' aa3gp 
rcl ’ ,  ’ aa3fgrcl '  ,  ’ aa3fprcl ’ ,  1 aa3pgrcl ’ ,  ' aa3pf rcl 1 ) 

aa3gorrcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
aa3f orrcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
aa3porrcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
aa3gf orrcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
aa3gporrcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
aa3f gorrcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
aa3fporrcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
aa3pgorrcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
aa3pf orrcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' C : \users\owner\desktop\A3 

f iles\a3data\aa3richclor 1 , 'aa3gorrcl', 'aa3forrcl', 'aa3porrcl', 'aa3gforr 
cl  '  ,  ' aa3gporrcl 1 ,  ' aa3fgorrcl '  ,  1 aa3fporrcl 1 ,  1 aa3pgorrcl ' ,  ' aa3pforrcl 1 ) 

%  Target  deficient  environment 
load ( ' apdef a3 ’ ) 

aa3gdcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
aa3f dcl=roc (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
aa3pdcl=roc (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
aa3gfdcl=roc (pet 12 , pen 12 , pet23, pen23, pet, pen, i, fpa, tpag, fpbf, tpbf) ; 
aa3gpdcl=roc (pet 12 , pen 12 , pet23, pen23, pet, pen, i, fpa, tpag, fpbp, tpbp) ; 
aa3fgdcl=roc (pet 12 , pen 12 , pet23, pen23, pet, pen, i, fpa, tpaf, fpbg, tpbg) ; 
aa3fpdcl=roc (pet 12 , pen 12 , pet23, pen23, pet, pen, i, fpa, tpaf, fpbp, tpbp) ; 
aa3pgdcl=roc (pet 12 , pen 12 , pet23, pen23, pet, pen, i, fpa, tpap, fpbg, tpbg) ; 
aa3pfdcl=roc (pet 12 , pen 12 , pet23, pen23, pet, pen, i, fpa, tpap, fpbf, tpbf) ; 
save ( ' C : \users\owner\desktop\A3 

f iles\a3data\aa3def cl ' , ' aa3gdcl ' , ' aa3fdcl ' , ' aa3pdcl 1 , ' aa3gfdcl ' , ' aa3gpd 
cl ' ,  ' aa3fgdcl '  ,  ' aa3fpdcl ’ ,  ! aa3pgdcl ' ,  ’ aa3pfdcl ’ ) 

aa3gordcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpag, fpbg, tpbg) ; 
aa3f ordcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpaf, fpbf, tpbf) ; 
aa3pordcl=rocor (pet 12 , pen 12 , pet 2 3 , pen23 , pet, pen, i , fpa, tpap, fpbp, tpbp) ; 
aa3gf ordcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbf, tpbf) ; 
aa3gpordcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpag, fpbp, tpbp) ; 
aa3f gordcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf, fpbg, tpbg) ; 
aa3fpordcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpaf, fpbp, tpbp) ; 
aa3pgordcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbg, tpbg) ; 
aa3pf ordcl=rocor (pet 12 , pen 12 , pet23 , pen23 , pet, pen, i , fpa, tpap, fpbf, tpbf) ; 
save ( ' C : \users\owner\desktop\A3 

f iles\a3data\aa3def clor ' , 'aa3gordcl', ' aa3fordcl', 'aa3pordcl', 'aa3gfordc 
1 ! ,  ! aa3gpordcl ' ,  ' aa3fgordcl '  ,  ' aa3fpordcl 1 ,  ' aa3pgordcl ' ,  1 aa3pfordcl ! ) 
end 
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function  [roc]  =  roc (a, b, c, d, e ,  f ,  q, w, x, y, z ) 

%  roc  function  for  across  1/  across  III  AND.  Note  the  only  difference 
is 

%  the  prior  target  weightings, 
if  g==l 

rho=-0 . 8 ; 
else  if  g==2 

rho=-0 . 5 / 
else  if  g==3 

rho=-0 . 3 ; 
else  if  g==4 

rho=0 ; 

else  if  g==5 

rho=0 . 3 ; 
else  if  g==6 

rho=0 . 5 ; 
else  if  g==7 

rho=0 . 8 ; 

end 

end 

end 

end 

end 

end 

end 

n=length (w) ; 
m=length (y) ; 
for  i=l : n 

for  j  =1 : m 

fpand  (i,  j  )=w(i)*y(j); 
costA ( i ) =a*x ( i ) +b^w ( i ) ; 
costB  ( j ) =c*z ( j ) +d*y ( j ) ; 

corr (if  j ) =sqrt (costA (i) * (1-cost A (i) ) ) *sqrt (costB (j ) * (1- 
costB ( j ) ) ) ; 

tpand (i, j ) - (1/e) * (rho*corr (if  j ) +costA (i) ^ costB ( j ) - 
f^ fpand (ir  j ) ) ; 
end 

end 

improc=s or trows ( cat ( 2 ,  fpand ( : ) ,  tpand ( : ) ) , 1 ) ; 
c2=improc (: ,2) ; 
btp=f ind (c2<0 ) ; 
if  isempty (btp) ~=1  ; 
c2 (btp) =0 ; 

end 

ind=f ind ( c2>l )  ; 
if  isempty ( ind) ~=1 ; 
c2 ( ind) =1 ; 
end 

improc=cat ( 2 , improc ( : , 1 ) ,  c2 ) ; 
c2=improc ( : , 2 ) ; 
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q=length  ( c2 )  ; 

dmat= [ 0 ; c2  (1:  (q-1) )  ]  ; 

dif f=c2-dmat ; 

cl=improc  (:,1)  ; 
nf=find(diff<0) ; 
n=length (nf ) ; 

i=l  ; 

while  i<=n 

improc (nf (i ) ,  : ) =cat ( 2 , cl (nf (i) ) ,c2 (nf (i) -1) ) ; 
i=i  +  l ; 

end 

roc=unique (round (improc* 10000) /10000, ’ rows ’ ) ; 

i=l; 

while  i<=length ( roc) 
if  rocheck ( roc) ~=1 ; 

roc=cloop (roc) ; 
else  if  rocheck (roc) ==1 ; 
break 

end 

end 
i=i+l ; 
end 

vfp=roc  (  : , 1 )  ; 
vtp=roc  (: ,2)  ; 

for  i=l : length (vfp) 

ind=f ind ( vfp==vfp ( i ) ) / 
tpmax (i) =max (vtp (ind) ) ; 
fpmin (i) =min (vfp (ind)  )  ; 

end 

roc=unique ( cat ( 2 , fpmin ' ,  tpmax ’ ) ,  ' rows ’ ) ; 
end 

function  [rocor]  =  rocor (a, b, c, d^ e ,  f , q, w, x, y, z ) 

%  function  that  produces  across  1/  across  III  OR  ROC  curves 
if  g==l 

rho=-0 . 8 ; 
else  if  g==2 

rho=-0 . 5 ; 
else  if  g==3 

rho=-0 . 3 ; 
else  if  g==4 

rho=0 ; 

else  if  g==5 

rho=0 . 3 ; 
else  if  g==6 

rho=0 . 5 ; 
else  if  g==7 

rho=0 . 8 ; 
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end 


end 

end 

end 

end 

end 

end 

n=length (w) ; 
m=length (y) ; 
for  i=l : n 

for  j  =1 : m 

fpor  (i,j)=w(i)+y(j)-w(i)*y(j); 
costA ( i ) =a*x ( i ) +b*w ( i ) ; 
costB (j ) =c*z ( j ) +d*y ( j ) ; 

corr (i, j ) =sqrt (costA (i) * (1-cost A (i) ) ) *sqrt (costB ( j ) * (1- 
costB  ( j ) ) )  / 

tpor (i, j ) = (1/e) * (costA (i) +costB ( j ) -rho*corr (i, j ) - 
(costA (i) * costB ( j ) ) -f *fpor (i, j ) ) ; 
end 

end 

improc=s or trows (cat (2, fpor ( : ) ,  tpor ( : ) ) , 1 ) ; 
c2=improc ( : f  2 ) ; 
btp=f ind ( c2<0 ) / 
if  isempty (btp) ~=1 ; 
c2 (btp) =0 ; 

end 

ind=f ind ( c2>l )  ; 
if  isempty ( ind) ~=1 ; 
c2 ( ind) =1 ; 
end 

improc=cat ( 2 , improc ( : , 1 )  ,  c2 )  ; 

c2=improc ( : , 2 ) ; 
q=length (c2 ) ; 
dmat= [ 0 ; c2 ( 1 : (q-1))]; 
dif f=c2-dmat ; 

cl=improc  ( : , 1 )  ; 
nf=f ind (dif f<0 ) ; 
n=length (nf ) ; 

i=l; 

while  i<=n 

improc (nf (i) r  : ) =cat (2, cl (nf (i) ) , c2 (nf (i) -1) ) ; 
i=i+l ; 

end 

rocor=unique (round (improc^ 10000) /10000, ' rows ’ ) ; 
i=l  ; 

while  i<=length ( rocor ) 
if  rocheck ( rocor ) ~=1 ; 
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rocor=cloop (rocor ) ; 
else  if  rocheck (rocor ) ==1 ; 
break 

end 

end 
i=i+l ; 
end 

vfp=rocor  (:,1)  ; 
vtp=rocor  (: ,2)  ; 

for  i=l : length (vfp) 

ind=find (vfp==vfp (i) ) ; 
tpmax (i) =max (vtp (ind) ) ; 
fpmin (i) =min (vfp (ind)  )  ; 

end 

rocor=unique ( cat ( 2 ,  fpmin ’ , tpmax ' ) ,  ' rows ' ) ; 
end 

function  [bias]  =  bias(x,y) 

%Bias  between  within  and  across  classifiers. 

%cl  and  c2  are  the  false  positive  and  true  positive  rates  for  the  ROC 
curve 

%of  the  across  classifier 

%c3  and  c4  are  the  fpr  and  tpr  for  the  ROC  curve  of  the  within 

classifier 

cl— x (:,1); 

c2=x  (  : , 2 )  ; 

c3=y (  :  ,  1)  ; 

c4=y ( : , 2)  ; 

%  Returns  the  indices  of  those  fpr  values  that  lie  in  the  intersection 
of 

%  the  fpr  for  the  across  classifier  and  the  fpr  for  the  within 
classifier 

[ r , a, w] ^intersect (cl, c3) ; 

adata=cat (2 ,  r , c2 (a) ) ;  %  fpr  and  tpr  for  across  classifier 
wdata=cat ( 2 , r , c4 (w) ) ;  %  fpr  and  tpr  for  within  classifier 
bias=cat (2 , r , wdata ( : , 2 ) -adata ( : , 2 ) ) ;  %  bias  =  difference  in  tpr  at 
given  fpr 
end 

function  [negbias]  =  negbias (a,b,c,d,e, f, s,h,q, r) 

%  Returns  graphs  for  non-correlated  and  all  negitively  correlated  ROC 
%  curves . 

g=bias(a,b);  %non-correlated 
g2=bias (c, d) ;  %cl 
g3=bias (e , f ) ;  %c2 
g4=bias ( s f h) ;  %c3 

negb i a s= figure ( 'visible ' ,  ' of f ' ) ; 
hold  on 

plot (g ( : , 1) , g  ( : , 2) ) 

plot  (g2  ( : ,  1 )  ,  g2  ( : ,  2 )  ,  '  —  ') 
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plot(g3(:,l)  ,g3(:,2)  ,  '  :  ’) 
plot(g4(:,l),g4(:,2),  ’x’) 
xlabel ( ' max  pq=r ' ) 

ylabel('bias  (difference  in  tpr) '  ) 
title (q) 

saveas (negbias,  r) 
end 

function  [negbiasor]  =  negbiasor (a,b,c,d,e, f, s,h,q,r) 

%  Returns  graphs  for  non-correlated  and  all  negitively  correlated  ROC 
%  curves . 

g=bias(a,b);  %non-correlated 
g2=bias (c, d) ;  %cl 
g3=bias (e, f ) ;  %c2 
g4=bias ( s , h) ;  %c3 

negbiasor=f igure ( ’visible ' ,  ' of f ' ) ; 
hold  on 

plot  (g  ( : ,  1)  ,  g  ( : ,  2)  ) 

plot  (g2  ( : ,  1 )  ,  g2  ( : ,  2 )  ,  •  — •) 

plot (g3 ( : , 1 ) , g3 ( : , 2 ) ,  '  : 

plot (g4(:,l),g4(:,2), 'x ' ) 

xlabel ( ’ max  p+q-pq=r ' ) 

ylabel('bias  (difference  in  tpr) ') 

title (q) 

saveas (negbiasor,  r) 
end 

function  [posbias]  =  posbias (a, b, c, d, e, f , s, h, q, r) 

%  Returns  graphs  for  non-correlated  and  all  positively  correlated  ROC 
%  curves. 

g=bias(a,b);  %non-correlated 
g2=bias (c, d) ;  %cl 
g3=bias (e , f ) ;  %c2 
g4=bias ( s , h) ;  %c3 

po sb i a s= figure (' visible ' ,  ’ of f ’  )  ; 
hold  on 

plot  (g  ( : ,  1)  ,  g  ( : ,  2)  ) 
plot  (g2  ( : ,  1 )  ,  g2  ( : ,  2 )  ,  '  —  ') 
plot (g3 ( : , 1 ) , g3 ( : , 2 ) ,  '  :  ') 
plot (g4(:,l),g4(:,2), 'x ' ) 
xlabel ( ' max  pq=r ' ) 

ylabel('bias  (difference  in  tpr) ') 
title (q) 

saveas (posbias,  r) 
end 

function  [posbiasor]  =  posbiasor (a, b, c, d, e, f , s, h, q, r) 

%  Returns  graphs  for  non-correlated  and  all  positively  correlated  ROC 
%  curves. 

g=bias(a,b);  %non-correlated 
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g2=bias (c, d) ;  %cl 
g3=bias (e ,  f ) ;  %c2 
g4=bias ( s , h) ;  %c3 

posbiasor=f igure (  ' visible ’ ,  ' of f ' ) ; 
hold  on 

plot  (g  ( : ,  1)  ,  g  ( : ,  2)  ) 

plot  (g2  ( : ,  1 )  ,  g2  ( : ,  2 )  ,  '  —  ') 

plot (g3 ( : , 1 ) , g3 ( : , 2 ) ,  '  : 

plot (g4 ( : , 1) , g4 ( : , 2) , 'x' ) 

xlabel ( ’ max  p+q-pq=r ' ) 

ylabel ('bias  (difference  in  tpr) ') 

title (q) 

saveas (posbiasor,  r) 
end 


For  brevity,  only  one  section  of  the  figures  code  is  included  here.  All  other  codes  are 
approximately  the  same  except  for  changes  in  variables  and  strings. 


clear; clc 

%%  Enhanced  figures  alt  prev 

load ( ’ alen ’ ) ; 

load ( ’ Bf ront ’ ) ; 

load ( ’ Af ront ’ ) ; 

load ( ’ alenor ’ ) ; 

load ( ' wen ’ ) ; 

load ( ' wenor ’ ) ; 

load ( ’ a2en ’ ) ; 

load ( ’ a2enor ’ ) ; 

load ( ’ a3en ' ) ; 

load ( ’ a3enor ’ )  ; 

%%  A  vs  B  I  &  III 
AvBg= figure ( 1  visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, ag) 

plot  (ubg  (  :  A  1 )  r  ubg  (  :  ,  2 )  f  —  ’) 

xlabel ( ! Classifier  A  Good  performance  =  line') 
ylabel ( ! Classifier  B  Good  performance  =  dash') 
title ('A  vs  B  "good"') 

saveas (AvBg, ' c : \users\owner\desktop\graphs\ETP\A  vs  B  G.jpg') 

AvBf=f igure ( ' visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, af ) 

plot  (ubf ( : , 1 ) , ubf ( :  ,  2 )  ,  ’  --  ’  ) 

xlabel ( ! Classifier  A  Good  performance  =  line') 
ylabel (’ Classifier  B  Good  performance  =  dash') 
title ('A  vs  B  "fair"') 

saveas (AvBf, ' c : \users\owner\desktop\graphs\ETP\A  vs  B  F.jpg') 

AvBp= figure ( ' visible '  ,  ' of f ' )  ; 
hold  on 
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plot  ( fpa, ap) 

plot  (ubp  (  :  ,  1 )  ,  ubp  (  :  ,  2 ) 

xlabel (' Classifier  A  Good  performance  =  line') 
ylabel ( ! Classifier  B  Good  performance  =  dash') 
title ('A  vs  B  "poor"') 

saveas (AvBp, ' c : \users\owner\desktop\graphs\ETP\A  vs  B  P.jpg') 

%%  Across  I  vs  within  ROC 
Gl  =  f igure ( 'visible '  ,  ' of f ' ) ; 
hold  on 
plot ( fpa, ag) 

plot (ubg ( : , 1 ) , ubg ( : , 2 ) ,  '  —  '  ) 
plot (alg(:,l) ,alg(:,2) ,  '  : 
plot (wg ( : , 1 ) , wg ( : , 2 ) , ' * ' ) 
xlabel ('A  =  solid,  B  =  dash') 
ylabel ('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  "good"  vs  within  "good"') 

saveas (Gl,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  G  vs 
within  G . j pg ' ) 

Fl  =  f igure ( 'visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, af ) 

plot  (ubf ( : , 1 ) , ubf ( :  ,  2 )  ,  '  --  '  ) 
plot  (alf ( : , 1) ,alf (:,2)  ,  '  :  ') 
plot ( wf ( : , 1 ) ,wf (:,2)  ,  '*') 
xlabel ('A  =  solid,  B  =  dash') 
ylabel ('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  "fair"  vs  within  "fair"') 

saveas (FI,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  F  vs 
within  F . jpg ' ) 

PI  =  f igure ( 'visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, ap) 

plot (ubp ( : , 1 ) , ubp  (  : , 2 ) ,  '  —  '  ) 
plot (alp ( : , 1) , alp  (  : , 2) ,  '  :  '  ) 
plot (wp ( : , 1 ) , wp  (  : , 2 ) ,  ' *  '  ) 
xlabel ('A  =  solid,  B  =  dash') 
ylabel ('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  "poor"  vs  within  "poor"') 

saveas (PI,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  P  vs 
within  P . j  pg ' ) 

gf l=f igure ( 'visible ' , ' of f ' ) ; 
hold  on 
plot ( fpa, ag) 

plot  (ubf ( : , 1) , ubf ( :  ,  2)  ,  '  --  '  ) 
plot  (algf ( : , 1) , algf ( : , 2)  ,  '  :  '  ) 
plot (wgf ( : , 1) , wgf ( :  ,  2)  ,  ' * '  ) 
xlabel ('A  =  solid,  B  =  dash') 
ylabel ('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  A=G  B=F  vs  within  A=G  B=F ' ) 
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saveas(gfl,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  A=G 
B=F  vs  within  A=G  B=F.jpg') 

gpl=f igure ( 'visible ' ,  ' of f ' ) ; 
hold  on 
plot ( fpa, ag) 

plot  (ubp (:,1), ubp ( :  ,  2 )  ,  '  —  '  ) 
plot  (algp (:,1), algp ( : , 2)  ,  '  :  '  ) 
plot (wgp (:,1), wgp ( :  ,  2 )  ,  ' * '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  A =G  B=P  vs  within  A =G  B=P ' ) 

saveas(gpl,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  A=G 
B=P  vs  within  A=G  B=P.jpg') 

fgl=f igure ( 'visible ' , ' of f ' ) ; 
hold  on 
plot ( fpa, af ) 

plot  (ubg ( : , 1 ) , ubg ( :  ,  2 )  ,  '  —  '  ) 
plot  (alfg ( : , 1) , alfg ( :  ,  2)  ,  '  :  '  ) 
plot  (wfg ( : , 1) , wfg ( :  ,  2)  ,  ' * '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  A=F  B=G  vs  within  A=F  B=G ' ) 

saveas(fgl,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  A=F 
B=G  vs  within  A=F  B=G.jpg') 

fpl  =  f igure ( 'visible ' ,  1  of f ' )  ; 
hold  on 
plot ( fpa, af ) 

plot (ubp ( : , 1 ) , ubp  (  : , 2 ) ,  '  —  '  ) 
plot (alfp ( : , 1) , alfp  (  : , 2) ,  '  :  '  ) 
plot (wfp ( : , 1) , wfp  (  : , 2) ,  ' *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  A=F  B=P  vs  within  A=F  B=P ' ) 

saveas(fpl,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  A=F 
B=P  vs  within  A=F  B=P.jpg') 

pgl  =  f igure (' visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, ap) 

plot  (ubg  ( :  ,  1 )  ,  ubg  ( :  ,  2 )  ,  '  — ') 
plot (alpg ( : , 1) , alpg ( : , 2) , ' : ' ) 
plot (wpg ( : , 1 ) , wpg ( : , 2 ) , ' * ' ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  A=P  B=G  vs  within  A=P  B=G ' ) 

saveas(pgl,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  A=P 
B=G  vs  within  A=P  B=G.jpg') 

pfl=f igure ( 'visible '  ,  1  of f ' ) ; 
hold  on 
plot ( fpa, ap) 
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plot (ubf ( : , 1) ,ubf  (:,2) ,  ’  —  '  ) 
plot (alpf (:,1) ,alpf  (:,2) ,  '  :  ') 
plot (wpf ( : , 1 ) , wpf ( : , 2 ) , ' * ’ ) 
xlabel ( ’ A  =  solids  B  =  dash') 
ylabel('Al  =  double  dot,  w  =  star’) 

title ('A  vs  B  vs  Across  I  A=P  B=F  vs  within  A=P  B=F 1 ) 

saveas (pfl,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  A=P 
B=F  vs  within  A=P  B=F.jpg') 

%%  ROC  Across  II  vs  within  AND 
G2=f igure ( 'visible ' , ' of f ' ) ; 
hold  on 
plot ( fpa, ag) 

plot (ubg ( : , 1 ) , ubg  ( : , 2 ) ,  '  —  '  ) 
plot (a2g ( : , 1 ) , a2g ( : , 2 ) ,  '  : 
plot (wg ( : , 1 ) , wg ( : , 2 ) ,  ' *  ' ) 
xlabel ('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  II  "good"  vs  within  "good"') 

saveas (G2,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II  G  vs 
within  G . j pg ' ) 

F2  =  f igure ( 'visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, af ) 

plot  (ubf ( : , 1) , ubf ( :  ,  2)  ,  '  --  '  ) 
plot  (a2f ( : , 1) , a2f ( : , 2 )  ,  '  :  ') 
plot (wf ( : , 1) ,wf (:,2)  ,  '*') 
xlabel ('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  II  "fair"  vs  within  "fair"') 

saveas (F2,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II  F  vs 
within  F . j  pg ' ) 

P2  =  f igure ( 'visible '  ,  ' of f ' ) ; 
hold  on 
plot ( fpa, ap) 

plot (ubp ( : , 1 ) , ubp  (  : , 2 ) ,  '  —  '  ) 
plot (a2p(:,l) ,a2p(:,2) ,  '  :  ') 
plot (wp ( : , 1 ) , wp ( : , 2 ) ,  ' *  '  ) 
xlabel ('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  II  "poor"  vs  within  "poor"') 

saveas (P2,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II  P  vs 
within  P . jpg ' ) 

GF2  =  f igure (' visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, ag) 

plot  (ubf ( : , 1 ) , ubf ( :  ,  2 )  ,  '  --  '  ) 
plot  (a2gf ( : , 1) ,a2gf ( : ,2)  ,  '  :  ') 
plot (wgf ( : , 1) , wgf ( :  ,  2)  ,  ' * '  ) 
xlabel ('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 
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title ('A  vs  B  vs  Across  II  A=G  B=F  vs  within  A=G  B=F ’ ) 

saveas (GF2,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II  A=G 
B=F  vs  within  A=G  B=F.jpg') 

GP2=f igure ( 'visible ' ,  ' of f ' ) ; 
hold  on 
plot ( fpa, ag) 

plot  (ubp  (  :  ,  1 )  ,  ubp  (  :  ,  2 )  ,  '  — 
plot (a2gp ( : , 1 ) , a2gp ( : ,  2 )  ,  ' : '  ) 
plot  (wgp (: ,1) , wgp ( : ,  2 )  ,  ' * ' ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  II  A=G  B=P  vs  within  A=G  B=P ' ) 

saveas (GP2,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II  A=G 
B=P  vs  within  A=G  B=P.jpg') 

FG2=f igure (' visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, af ) 

plot  (ubg ( : , 1) , ubg ( :  ,  2)  ,  '  —  '  ) 
plot  (a2fg(:,l) ,a2fg(:,2)  ,  '  :  ') 
plot (wfg ( : , 1) , wfg ( : , 2)  ,  ' * '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  II  A=F  B=G  vs  within  A=F  B=G ' ) 

saveas (FG2,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II  A=F 
B=G  vs  within  A=F  B=G.jpg') 

FP2  =  f igure ( 'visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, af ) 

plot (ubp ( : , 1 ) , ubp  (  : , 2 ) ,  '  —  '  ) 
plot (a2fp ( : , 1) , a2fp  (  : , 2) ,  '  :  '  ) 
plot (wfp ( : , 1) , wfp  (  : , 2) ,  '  *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  II  A=F  B=P  vs  within  A=F  B=P ' ) 

saveas (FP2,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II  A=F 
B=P  vs  within  A=F  B=P.jpg') 

PG2=f igure (' visible ' , ' of f ' ) ; 
hold  on 
plot ( fpa, ap) 

plot (ubg ( : , 1 ) , ubg ( : , 2 ) ,  '  —  '  ) 
plot (a2pg ( : , 1) , a2pg  (  : , 2) ,  '  :  '  ) 
plot (wpg ( : , 1 ) , wpg ( : , 2 ) , ' * ' ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  II  A=P  B=G  vs  within  A=P  B=G ' ) 

saveas (PG2,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II  A=P 
B=G  vs  within  A=P  B=G.jpg') 

PF2=f igure (' visible ' , ' of f ' ) ; 
hold  on 
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plot  ( fpa, ap) 

plot  (ubf ( : , 1) , ubf ( : , 2) ,  ’ -- '  ) 
plot  (a2pf ( : ,  1) ,  a2p f ( :  ,  2 )  ,  '  :  ') 
plot (wpf ( : , 1 ) , wpf ( :  ,  2 )  ,  ’  *  ’  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star’) 

title ('A  vs  B  vs  Across  II  A=P  B=F  vs  within  A=P  B=F ’ ) 

saveas(PF2,  ’c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II  A=P 
B=F  vs  within  A=P  B=F.jpg') 

%%  Across  3  vs  within  ROC 
G3= figure ( 'visible ' , ' of f ' ) ; 
hold  on 
plot ( fpa, ag) 

plot  (ubg  ( :  ,  1 )  ,  ubg  ( :  ,  2 )  ,  '  — 
plot (a3g ( : , 1) , a3g ( : , 2) ,  '  :  '  ) 
plot (wg ( : , 1 ) , wg  (  : , 2 ) ,  ' *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  "good"  vs  within  "good"') 

saveas (G3,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III  G  vs 
within  G . j pg ' ) 

F3= figure ( 'visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, af ) 

plot  (ubf ( : , 1) , ubf ( :  ,  2)  ,  '  --  '  ) 
plot  (a3f ( : , 1) , a3f ( : , 2 )  ,  '  :  ') 
plot (wf ( : , 1) ,wf (:,2)  ,  '*') 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  "fair"  vs  within  "fair"') 

saveas (F3,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III  F  vs 
within  F . j  pg ' ) 

P3= figure ( 'visible '  ,  ' of f ' ) ; 
hold  on 
plot ( fpa, ap) 

plot  (ubp  ( :  ,  1 )  ,  ubp  ( :  ,  2 )  ,  '  — ') 
plot (a3p(:,l) ,a3p(:,2) ,  '  :  ') 
plot (wp ( : , 1 ) , wp  (  : , 2 ) ,  ' *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  "poor"  vs  within  "poor"') 

saveas (P3,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III  P  vs 
within  P . j  pg ' ) 

gf 3= figure ( 'visible '  ,  ' of f ' ) ; 
hold  on 
plot ( fpa, ag) 

plot  (ubf ( : , 1) , ubf ( :  ,  2)  ,  '  --  '  ) 
plot  (a3gf ( : , 1) ,a3gf ( : ,2)  ,  '  :  ') 
plot (wgf ( : , 1) , wgf ( :  ,  2)  ,  ' *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
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ylabel('A3  =  double  dot,  w  =  star’) 

title ('A  vs  B  vs  Across  III  A=G  B=F  vs  within  A=G  B=F ' ) 

saveas (gf3,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III  A=G 
B=F  vs  within  A=G  B=F.jpg') 

gp 3= figure ( 'visible ' ,  *  of f ' ) ; 
hold  on 
plot  ( fpa, ag) 

plot  ( ubp  (  :  ,  1 )  ,  ubp  (  :  ,  2 )  ,  '  — ') 
plot (a3gp ( : , 1 ) , a3gp ( : ,  2 )  ,  ' : '  ) 
plot (wgp ( : , 1 ) , wgp ( :  ,  2 )  ,  ' * '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  A=G  B=P  vs  within  A =G  B=P ' ) 

saveas (gp3,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III  A =G 
B=P  vs  within  A=G  B=P.jpg') 

fg3= figure ( 'visible ' , ’off' ) ; 
hold  on 
plot  ( fpa, af ) 

plot  (ubg  ( :  ,  1 )  ,  ubg  ( :  ,  2 )  ,  '  — ') 
plot (a3fg(:,l) ,a3fg(:,2) , ' : ') 
plot (wfg ( : , 1) , wfg ( : , 2) , ' * ' ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  A=F  B=G  vs  within  A=F  B=G ' ) 

saveas (fg3,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III  A=F 
B=G  vs  within  A=F  B=G.jpg') 

fp 3= figure ( 'visible ' , ' of f ' ) ; 
hold  on 
plot ( fpa, af ) 

plot (ubp ( : , 1 ) , ubp  (  : , 2 ) ,  '  —  '  ) 
plot (a3fp(:,l) ,a3fp(:,2) ,  '  :  ') 
plot (wfp ( : , 1) , wfp  (  : , 2) ,  ' * ' ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  A=F  B=P  vs  within  A=F  B=P ' ) 

saveas (fp3,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III  A=F 
B=P  vs  within  A=F  B=P.jpg') 

pg3= figure (' visible '  ,  ' of f ' ) ; 
hold  on 
plot ( fpa, ap) 

plot (ubg ( : , 1 ) , ubg (:, 2 ),'--'  ) 
plot (a3pg ( : , 1) , a3pg ( : , 2) , ' : ' ) 
plot (wpg ( : , 1 ) , wpg ( : , 2 ) , ' * ' ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  A=P  B=G  vs  within  A=P  B=G ' ) 

saveas (pg3,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III  A=P 
B=G  vs  within  A=P  B=G.jpg') 

pf 3= figure ( 'visible ' , ' of f ' ) ; 
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hold  on 
plot ( fpa, ap) 

plot  (ubf ( : , 1 ) , ubf ( : , 2 ) ,  '  -  -  '  ) 
plot (a3pf ( : , 1 ) , a3pf  (  : ,  2 ) ,  ’  :  ’  ) 
plot (wpf ( : , 1 ) , wpf ( : , 2 ) ,  ' *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  A=P  B=F  vs  within  A=P  B=F '  ) 

saveas (pf3,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III  A=P 
B=F  vs  within  A=P  B=F.jpg') 

%%  ROC  Across  I  vs  within  OR 
Glor=f igure ( 'visible ' , ' of f ' ) ; 
hold  on 
plot ( fpa, ag) 

plot  (ubg ( : , 1 ) , ubg ( : , 2 )  ,  '  —  '  ) 
plot  (algor (:, 1 ) , algor ( :  ,  2 )  ,  '  :  '  ) 
plot (wgor ( : , 1 ) , wgor ( :  ,  2 )  ,  '  *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  "good"  vs  within  "good"  OR') 

saveas (Glor,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  G  vs 
within  G  OR. jpg') 

FI or=f igure ( 'visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, af ) 

plot (ubf ( : , 1 ) , ubf ( : , 2 ) , ' -- ' ) 
plot (alf or ( : , 1 ) , alf or  (  : , 2 ) ,  '  :  ') 
plot (wf or ( : , 1 ) , wf or ( : , 2 ) , ' * ' ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  "fair"  vs  within  "fair"  OR') 

saveas (Flor,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  F  vs 
within  F  OR. jpg') 

PI or=f igure ( 'visible '  ,  ' of f ' ) ; 
hold  on 
plot ( fpa, ap) 

plot  (ubp  ( : , 1 ) , ubp ( : , 2 ) ,  '--') 
plot  (alpor ( : , 1 ) , alpor ( :  ,  2 )  ,  '  :  '  ) 
plot  (wpor ( : , 1 ) , wpor ( :  ,  2 )  ,  '  *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  "poor"  vs  within  "poor"  OR') 

saveas (Plor,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  P  vs 
within  P  OR. jpg') 

gflor=f igure ( 'visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, ag) 

plot  (ubf ( : , 1) , ubf ( :  ,  2)  ,  '  --  '  ) 
plot(algfor(:,l) ,algfor(:,2)  ,  '  :  ') 
plot (wgf or ( : , 1 ) , wgf or ( :  ,  2 )  ,  '  *  '  ) 
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xlabel('A  =  solid,  B  =  dash') 
ylabel('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  A=G  B=F  vs  within  A=G  B=F  OR') 

saveas (gflor,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  A=G 
B=F  vs  within  A=G  B=F  OR. jpg') 

gplor=f igure ( 'visible '  ,  ' of f ' )  ; 
hold  on 
plot  ( fpa, ag) 

plot  (ubp  ( :  ,  1 )  ,  ubp  ( :  ,  2 )  ,  '  — 
plot (algpor ( : , 1 ) , algpor  (  : , 2 ) ,  '  :  '  ) 
plot (wgpor ( : , 1 ) , wgpor  (  : , 2 ) ,  ' * ' ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  A=G  B=P  vs  within  A=G  B=P  OR') 

saveas (gplor,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  A=G 
B=P  vs  within  A=G  B=P  OR. jpg') 

fglor=f igure ( 'visible ' , ' of f ' ) ; 
hold  on 
plot ( fpa, af ) 

plot  (ubg  ( :  ,  1 )  ,  ubg  ( :  ,  2 )  ,  '  — ') 
plot (alfgor ( : , 1 ) , alfgor ( : , 2 ) , ' : ' ) 
plot (wf gor ( : , 1 ) , wf gor  (  : , 2 ) ,  ' * ' ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  A=F  B=G  vs  within  A=F  B=G  OR') 

saveas ( fglor,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  A=F 
B=G  vs  within  A=F  B=G  OR. jpg') 

fplor=f igure ( 'visible '  ,  ' of f ' ) ; 
hold  on 
plot ( fpa, af ) 

plot  (ubp  ( :  ,  1 )  ,  ubp  ( :  ,  2 )  ,  '  — ') 
plot (alfpor ( : , 1 ) , alfpor  (  : , 2 ) ,  '  :  '  ) 
plot (wfpor ( : , 1 ) , wfpor ( : , 2 ) , ' * ' ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  A=F  B=P  vs  within  A=F  B=P  OR') 

saveas (fplor,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  A=F 
B=P  vs  within  A=F  B=P  OR. jpg') 

pglor=f igure (' visible ' , ' of f ' ) ; 
hold  on 
plot ( fpa, ap) 

plot  (ubg  ( :  ,  1 )  ,  ubg  ( :  ,  2 )  ,  '  — ') 
plot  (alpgor ( : , 1) , alpgor ( :  ,  2)  ,  '  :  '  ) 
plot (wpgor ( : , 1 ) , wpgor ( :  ,  2 )  ,  '  *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  A=P  B=G  vs  within  A=P  B=G  OR') 

saveas (pglor,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  A=P 
B=G  vs  within  A=P  B=G  OR. jpg') 
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pf lor=f igure ( ’visible '  ,  ' of f ' ) ; 
hold  on 
plot ( fpa, ap) 

plot  (ubf ( : , 1 ) , ubf ( : , 2 )  ,  ’ ) 
plot(alpfor(:,l) , alpf or  (  : , 2 ) ,  '  :  ') 
plot (wpf or (  :  ,  1 )  , wpf or  (: ,2) ,  '  *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('Al  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  I  A=P  B=F  vs  within  A=P  B=F  OR') 

saveas (pflor,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  I  A=P 
B=F  vs  within  A=P  B=F  OR. jpg') 

%%  ROC  II  OR 

G2or=f igure ( 'visible ' , ' of f ' ) ; 
hold  on 
plot ( fpa, ag) 

plot  (ubg ( : , 1 ) , ubg ( :  ,  2 )  ,  '  —  '  ) 
plot(a2gor(:,l) ,a2gor(:,2)  ,  '  :  ') 
plot (wgor ( : , 1 ) , wgor ( :  ,  2 )  ,  '  *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  II  "good"  vs  within  "good"  OR') 

saveas (G2or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II  G 

vs  within  G  OR. jpg') 

F2or=f igure ( 'visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, af ) 

plot  (ubf ( : , 1 ) , ubf ( :  ,  2 )  ,  '  -  -  '  ) 
plot  (a2f or ( : , 1 ) , a2f or ( :  ,  2 )  ,  '  :  '  ) 
plot (wf or ( : , 1 ) , wf or ( :  ,  2 )  ,  '  *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  II  "fair"  vs  within  "fair"  OR') 

saveas (F2or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II  F 

vs  within  F  OR. jpg') 

P2or=f igure ( 'visible ' , ' of f ' ) ; 
hold  on 
plot ( fpa, ap) 

plot (ubp ( : , 1 ) , ubp  (  : , 2 ) ,  '  —  '  ) 
plot (a2por ( : , 1 ) , a2por  (  : , 2 ) ,  '  :  '  ) 
plot (wpor ( : , 1 ) , wpor ( : , 2 ) , '  *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  II  "poor"  vs  within  "poor"  OR') 

saveas (P2or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II  P 

vs  within  P  OR. jpg') 

gf 2or=f igure ( 'visible ' , ' of f ' ) ; 
hold  on 
plot ( fpa, ag) 

plot  (ubf ( : , 1) , ubf ( :  ,  2)  ,  '  --  '  ) 
plot  (a2gf or ( : , 1 ) , a2gf or ( :  ,  2 )  ,  '  :  '  ) 
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plot (wgf or (  :  ,  1 )  , wgf or ( : , 2 ) ,  ' *  ' ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  II  A=G  B=F  vs  within  A=G  B=F  OR’) 

saveas (gf2or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II 

A=G  B=F  vs  within  A=G  B=F  OR. jpg') 

gp2or=f igure ( ’visible ' , ' of f ' ) ; 
hold  on 
plot ( fpa, ag) 

plot (ubp ( : , 1) , ubp  (  : , 2) ,  '  —  '  ) 
plot (a2gpor (:,!), a2gpor  (  : , 2 ) ,  '  :  '  ) 
plot (wgpor ( : , 1 ) , wgpor ( : , 2 ) , ' * ' ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  II  A=G  B=P  vs  within  A=G  B=P  OR') 

saveas (gp2or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II 

A=G  B=P  vs  within  A=G  B=P  OR. jpg') 

fg2or=f igure ( 'visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, af ) 

plot  (ubg ( : , 1 ) , ubg ( :  ,  2 )  ,  '  —  '  ) 
plot  (a2f gor ( : , 1 ) , a2f gor ( :  ,  2 )  ,  '  :  '  ) 
plot (wf gor ( : , 1 ) , wf gor ( :  ,  2 )  ,  '  *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  II  A=F  B=G  vs  within  A=F  B=G  OR') 

saveas ( fg2or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II 

A=F  B=G  vs  within  A=F  B=G  OR. jpg') 

fp2or=f igure ( 'visible '  ,  ' of f ' ) ; 
hold  on 
plot ( fpa, af ) 

plot  (ubp  ( :  ,  1 )  ,  ubp  ( :  ,  2 )  ,  '  — ') 
plot (a2fpor ( : , 1 ) , a2fpor  (  : , 2 ) ,  '  :  '  ) 
plot (wfpor ( : , 1 ) , wfpor  (  : , 2 ) ,  ' * ' ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  II  A=F  B=P  vs  within  A=F  B=P  OR') 

saveas ( fp2or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II 

A=F  B=P  vs  within  A=F  B=P  OR. jpg') 

pg2or=f igure (' visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, ap) 

plot (ubg ( : , 1 ) , ubg ( : , 2 ) , '--') 
plot  (a2pgor ( : , 1) , a2pgor ( :  ,  2)  ,  '  :  '  ) 
plot  (wpgor ( : , 1 ) , wpgor ( :  ,  2 )  ,  '  *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  II  A=P  B=G  vs  within  A=P  B=G  OR') 

saveas (pg2or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II 

A=P  B=G  vs  within  A=P  B=G  OR. jpg') 
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pf 2or=f igure ( 1  visible  1 ,  ' of f ' ) ; 
hold  on 
plot ( fpa, ap) 

plot  (ubf  ( : , 1) , ubf ( : , 2) ,  ’  ) 

plot  (a2pf or (  :  ,  1 )  , a2pf or ( :  ,  2 )  ,  ’  :  ’  ) 
plot (wpf or ( : , 1 ) , wpf or ( :  ,  2 )  ,  '  *  ’  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A2  =  double  dot,  w  =  star’) 

title ('A  vs  B  vs  Across  II  A=P  B=F  vs  within  A=P  B=F  OR’) 

saveas (pf2or,  ’c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  II 

A=P  B=F  vs  within  A=P  B=F  OR. jpg') 

%%  Roc  Across  III  vs  within  OR 
G3or=f igure ( 'visible ' ,  ' of f '  )  ; 
hold  on 
plot ( fpa, ag) 

plot  (ubg ( : , 1 ) , ubg ( :  ,  2 )  ,  '  —  '  ) 
plot  (a3gor ( : , 1 ) , a3gor ( :  ,  2 )  ,  '  :  '  ) 
plot (wgor ( : , 1 ) , wgor ( :  ,  2 )  ,  ' *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  "good"  vs  within  "good"  OR') 

saveas (G3or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III  G 

vs  within  G  OR. jpg') 

F3or=f igure ( 'visible '  ,  ' of f ' )  ; 
hold  on 
plot ( fpa, af ) 

plot  (ubf ( : , 1) , ubf ( :  ,  2)  ,  '  --  '  ) 
plot  (a3f or ( : , 1 ) , a3f or ( :  ,  2 )  ,  '  :  ') 
plot (wf or ( : , 1 ) , wf or ( :  ,  2 )  ,  '  * '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  "fair"  vs  within  "fair"  OR') 

saveas (F3or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III  F 

vs  within  F  OR. jpg') 

P3or=f igure ( 'visible '  ,  ' of f ' ) ; 
hold  on 
plot ( fpa, ap) 

plot  (ubp  ( :  ,  1 )  ,  ubp  ( :  ,  2 )  ,  '  —  ') 
plot (a3por ( : , 1 ) , a3por  (  : , 2 ) ,  '  :  '  ) 
plot (wpor ( : , 1 ) , wpor ( : , 2 ) , '  *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  "poor"  vs  within  "poor"  OR') 

saveas (P3or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III  P 

vs  within  P  OR. jpg') 

gf 3or=f igure ( 'visible '  ,  ' of f ' ) ; 
hold  on 
plot ( fpa, ag) 

plot  (ubf ( : , 1) , ubf ( :  ,  2)  ,  '  --  '  ) 
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plot(a3gfor(:,l) ,a3gfor(:,2) ,  '  :  ’ ) 
plot (wgf or ( : , 1 ) , wgf or  (  : ,  2 ) ,  '  *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  A=G  B=F  vs  within  A=G  B=F  OR') 

saveas (gf3or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III 

A=G  B=F  vs  within  A=G  B=F  OR. jpg') 

gp3or=f igure ( ’visible '  ,  ' of f ' ) ; 
hold  on 
plot ( fpa, ag) 

plot (ubp ( : , 1 ) , ubp  (  : , 2 ) ,  '  —  '  ) 
plot (a3gpor ( : , 1 ) , a3gpor  ( : , 2 ) ,  ’  :  ’  ) 
plot (wgpor ( : , 1 ) , wgpor  (  : , 2 ) ,  '  *  '  ) 
xlabel(’A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  A=G  B=P  vs  within  A=G  B=P  OR') 

saveas (gp3or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III 

A=G  B=P  vs  within  A=G  B=P  OR. jpg') 

fg3or=f igure ( 'visible ' , ' of f ' ) ; 
hold  on 
plot ( fpa, af ) 

plot  (ubg ( : , 1 ) , ubg ( :  ,  2 )  ,  '  —  '  ) 
plot  (a3f gor ( : , 1 ) , a3f gor ( :  ,  2 )  ,  '  :  '  ) 
plot (wf gor ( : , 1 ) , wf gor ( :  ,  2 )  ,  ' * '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  A=F  B=G  vs  within  A=F  B=G  OR') 

saveas ( fg3or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III 

A=F  B=G  vs  within  A=F  B=G  OR. jpg') 

fp3or=f igure ( 'visible '  ,  ' of f ' ) ; 
hold  on 
plot ( fpa, af ) 

plot (ubp ( : , 1 ) , ubp  (  : , 2 ) ,  '  —  '  ) 
plot (a3fpor ( : , 1 ) , a3fpor  (  : , 2 ) ,  '  :  '  ) 
plot (wfpor ( : , 1 ) , wfpor ( : , 2 ) , ' * ' ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  A=F  B=P  vs  within  A=F  B=P  OR') 

saveas ( fp3or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III 

A=F  B=P  vs  within  A=F  B=P  OR. jpg') 

pg3or=f igure (' visible ' , ' of f ' ) ; 
hold  on 
plot ( fpa, ap) 

plot  (ubg  ( :  ,  1 )  ,  ubg  ( :  ,  2 )  ,  '  — ') 
plot  (a3pgor ( : , 1 ) , a3pgor ( :  ,  2 )  ,  '  :  '  ) 
plot (wpgor ( : , 1 ) , wpgor ( :  ,  2 )  ,  '  *  '  ) 
xlabel('A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  A=P  B=G  vs  within  A=P  B=G  OR') 
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saveas (pg3or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III 
A=P  B=G  vs  within  A=P  B=G  OR. jpg') 

pf 3or=f igure ( ’visible '  ,  ' of f ' ) ; 
hold  on 
plot ( fpa, ap) 

plot (ubf ( : , 1 ) , ub f  (  i  ,  2 ) ,  ' --  '  ) 
plot(a3pfor(:,l) , a3pf or  (  : , 2 ) ,  '  :  ') 
plot (wpf or ( : , 1 ) , wpf or  (  : ,  2 ) ,  ' *  ’  ) 
xlabel(’A  =  solid,  B  =  dash') 
ylabel('A3  =  double  dot,  w  =  star') 

title ('A  vs  B  vs  Across  III  A=P  B=G  vs  within  A=P  B=G  OR') 
title ('A  vs  B  vs  Across  III  A=P  B=F  vs  within  A=P  B=F  OR') 
saveas (pf3or,  'c:\users\owner\desktop\graphs\ETP\A  vs  B  vs  Across  III 
A=P  B=F  vs  within  A=P  B=F  OR. jpg') 
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